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I. Executive Summary 


This report presents the results from an evaluation of the fourth year of the Memphis Striving 
Readers Project (MSRP) conducted by Research for Better Schools (RBS). MSRP, funded for 
five years by the United States Department of Education, comprises two interventions aimed at 
improving adolescent literacy and the quality of literacy instruction across the curriculum. Eight 
Memphis City Schools (MCS) middle schools were chosen for inclusion in the study. 


The first intervention component, Scholastic’s READ 180 program (Enterprise Edition), has 
been implemented for four years in the MSRP schools. The 90-minute daily model includes 
instructional materials to be implemented as four 20-minute rotations in which students use the 
software program to build skills, engage in teacher-directed whole- and small-group instruction, 
read independently, and conclude with a 10-minute wrap-up. Students in the sixth through eighth 
grades who demonstrated the strongest need for reading support—i.e., who performed in the 
bottom quartile on the reading/language arts portion of a prior Tennessee Comprehensive 
Assessment Program (TCAP) test —were randomly selected to participate in the supplemental 
program or to serve as control group students. During Year 4, there were 809 students in the 
treatment (400) and control (409) groups. 


The second intervention component, the Memphis Content Literacy Academy (MCLA), is a 
whole-school professional development (PD) program designed by faculty, formerly at the 
University of Memphis, now at Bellarmine University, in collaboration with MCS staff. The 
overarching goal of MCLA is to improve literacy integration across the content areas through an 
intensive two-year PD program designed to develop middle school teachers’ use of literacy 
strategies to improve achievement in reading and in core content area subjects (defined as 
English language arts, mathematics, science and social studies). In addition to PD classes for 
teachers, MCLA included a course for school leaders, on-site coaching to support 
implementation, and an investment of $40,000 per school in supplemental instructional materials 
to be included in a Curriculum Resource Center (CRC). 


Researchers collected information about the implementation and impact of the two 
interventions using a variety of methods, including surveying, observing, and interviewing 
participating teachers as well as reviewing program documents and student scores on the 
Tennessee Comprehensive Assessment Program (TCAP) and Iowa Tests of Basic Skills (ITBS). 
A summary of key findings related to the implementation and impact of each intervention are 
presented next. 


READ 180 Implementation 


Analyses of data related to READ 180 implementation indicate implementation varies widely 
across the eight Striving Readers schools’ sixteen classrooms. However, implementation has 
improved each year. Percentages of classrooms that were rated as adequate with regard to 
overall implementation were lowest in Year 1 (slightly above 40%) and in Year 4 reached levels 
of more than 80 percent. Direct comparisons within the 12 classrooms that were unaffected by 
teacher attrition from Year 3 to Year 4 reveal the following: 
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* improved average Year 4 ratings, compared with average Year 3 ratings, of all components 
included in calculating fidelity (professional development, observations, and Scholastic 
Achievement Manager data) among 75 percent of classrooms 


* statistically significant school-level improvement regarding the number of READ 180 
sessions implemented per week, rising from 2.217 sessions in Year 3 to 2.752 in Year 4 
(although still below the recommended minimum of at least 3 sessions weekly)! 


* Significant student-level improvements in Year 4 compared to Year 3,including: 
- similar increases in the number of weekly sessions that were implemented 


- an average increase of more than 545 minutes using the software, reflecting reduced 
levels of technology-related problems 


READ 180 Impact 
A summary of READ 180 findings across the project’s four years follows: 


* No significant one-year impacts of participation in READ 180 were detected in Years 1, 2, 
or 4. 


* There was one small but significant one-year impact detected in Year 3 on sixth-grade 
students’ scores on the TCAP Reading/Language Arts test. 


* Analyses of interactions between the impacts of READ 180 and the whole-school 
intervention in Years | and 2 yielded no clearly interpretable patterns suggesting 
differences in the impact of READ 1/80 between MCLA treatment and control schools. 


* There were no significant two-year impacts of READ 180 Years 2, 3, or 4. 


* None of the analyses of combined ITT samples conducted in Year 4 yielded significant 
one- or two-year impacts. 


* The lack of findings related to Year 4 one- and two-year impacts was confirmed using a 
propensity score matching technique. 


MCLA Implementation 


Year 4 results indicate low to medium levels of MCLA implementation at each of the four 
Striving Readers schools. Implementation ratings were composed of the following components: 
levels of participation in MCLA, a coaching dosage score, a principal involvement rating, a use 
of materials score, and a teacher implementation score. In addition, RBS designed and 
facilitated a coaching retreat during which ratings of dimensions of teachers’ practice were 
assigned by coaches. Ratings were assigned based on descriptions of components explicated in 
the MCLA Innovation Configuration (IC) map. A cluster analysis showed that coaches rated 
over half (53%) of the 83 teachers as low-fidelity implementers, 26.5 percent as medium-fidelity 
implementers, and 20.5 percent as high-fidelity implementers. Findings suggest that the IC Map 


' Although the number of sessions per week remained below the minimum of three recommended by Scholastic, the 
relatively large number of minutes per session (18.48 minutes) indicates that students, on average, used the 
software for more than the minimum amount of time recommended over the course of the school year. 
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is a sufficiently sensitive framework to support empirical fidelity studies. Below are other 
findings related to Year 4 implementation: 


The amount of time respondents spent in staff development during Year 4 differed by 
cohort. Teachers who attended MCLA during the first two years (Cohort 1) reported 
significantly lower levels of professional development related to: (1) new teaching 
methods; (2) technology integration; (3) student assessment; (4) addressing English 
Language Learners; and (5) literacy integration compared to Cohort 2 teachers (those who 
participated in MCLA in Years 3 and 4). 


In Year 4, 42.5 hours of MCLA course-related professional development were offered. 
The remaining 47.5 hours (not documented) were to be completed through on-site work 
with coaches. Results show that, with the exception of science teachers, attendance by 
Cohort 2 participants was reasonably high. Other findings related to MCLA course-related 
participation are as follows: 

- The percentage of eligible teachers who enrolled in MCLA ranged from 44 percent to 
72.7 percent in the fall of Year 4 and dropped to between 24 percent and 60 percent by 
the spring. 

- 82.7 percent of all enrolled teachers completed both fall and spring semesters. 


According to coaching logs, 36.2 percent of all Year 4 MCLA participants received high 
levels of coaching support. 


Across time, the percentage of respondents who reported meeting with a literacy coach in 
the prior seven days increased. 


Principals reported feeling encouraged to conduct school walkthroughs with a focus on 
literacy, which they suggested strengthened their observational techniques and sparked 
discussions with teachers about instructional approaches. Despite increasing over time, 
only a quarter of teachers reported receiving administrator feedback, and this was the least 
frequently reported activity. 


Overall, ratings of teachers’ Curriculum Resource Center use were low. 


Significant increases were detected across time in percentages of teachers who reported 
formally assessing strategy use, meeting with a literacy coach, and receiving administrator 
feedback regarding literacy instruction. 


Observed use of vocabulary, fluency, and comprehension strategies during lessons was 
relatively even and ranged from 24 percent to 29 percent of strategies observed. 


MCLA Impact 


The experimental research design in Years | and 2 was a randomized matched-pairs design, 
with MCLA implemented in four of the eight schools. After the second year, MCLA ended in 
the experimental treatment schools, was revised to reflect lessons learned in Years 1 and 2, and 
moved to the four schools that initially served as control schools. Thus, the experimental phase 
of MCLA ended at the end of Year 2. Given that Year 4 represents completion of the second and 
final round of implementing MCLA, ordinary least squares regression (OLS) and a quasi- 
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experimental simulation analysis were carried out in an attempt to estimate improved 
achievement among students in the Cohort 2 schools that might be attributable to the whole- 
school intervention. Leveraging the design strength of using each school as its own “control,” 
achievement of students in the Cohort 2 schools at the end of Year 2 was compared to 
performance levels of students enrolled in these same schools at the end of Year 4 to estimate 
program effects. Main findings suggest the following: 


* The estimated impact of MCLA using OLS was not significant in any grade except eighth. 
- For eighth-grade students, Year 4 results were significantly better than in Year 2 for 
ITBS Total Reading and Vocabulary scores, with effect sizes approaching 0.20. 
- When the eighth-grade and seventh-grade students were combined, effect sizes were 
reduced and effects were not significant, despite the increased sample size. 


* The estimated impact of MCLA using the simulation approach indicated findings similar to 
those produced using OLS. The significance level and magnitude of effects on eighth- 
grade students’ ITBS Vocabulary and Total Reading scores are very close to the estimates 
derived from the OLS regression analyses. Specifically, two years of MCLA had overall 
treatment effects of 

- .24 standard deviations on eighth-grade students’ ITBS Vocabulary scores 

- .17 standard deviations on the eighth-grade students’ ITBS Total Reading scores 

- .09 standard deviations on the combined seventh- and eighth-grade students’ ITBS 
Vocabulary scores. 


* The two approaches produced consistent estimates of the magnitude of effects MCLA on 
seventh- and eighth-grade students’ vocabulary scores, although the estimated effect using 
OLS is not statistically significant. 


MSRP: Year 5 


This Year 4 report represents the final full report of the Memphis Striving Readers Project. 
During Year 5 (2010-2011), researchers at RBS and several project partners at Memphis City 
Schools and the University of Memphis began and will continue to focus on conducting 
additional analyses of data gathered and disseminating the findings of the project in a variety of 
media. Appendix ES comprises a list of these analysis and dissemination efforts. 
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ll. Introduction and Study Background 


Introduction 


This report presents the results from an evaluation of the fourth year of the Memphis Striving 
Readers Project (MSRP) conducted by Research for Better Schools (RBS). MSRP, funded for 
five years by the United States Department of Education, comprises two interventions aimed at 
improving adolescent literacy and the quality of literacy instruction across the curriculum. Eight 
Memphis City Schools (MCS) middle schools were chosen for inclusion in the study. Table 1 
summarizes the enrollment figures by schools for the four years of the MSRP and shows that 
over time, enrollment at the eight schools has decreased from 5,785 in Year 1 to 4,009 in Year 4. 


Table II-1: Student Enrollments in Schools Participating in the Striving Readers Study 


Memphis Middle Schools Enrollment in Enrollment in Enrollment in Enrollment in 
Participating in MSRP 2006-2007 2007-2008 2008-2009 2009-2010 
1 856 724 774 614 
2 997 1,034 1,021 989 
3 413 374 344 310 
4 635 520 462 387 
5 858 856 968 882 
6 640 603 539 406 
7 471 405 341 352 
8 915 759 713 683 
Total 5,785 5,275 5,162 4,009 


Data sources:http://edu.reportcard. state. tn.us/pls/apex/f?p=200: 1:3167232149886940 (Year 4) and 
http://www.memphis-schools.k12.tn.us/admin/communications/directoryofschools.html (Years 1-3) 

Note: A different data source was used in Year 4 because enrollment data were not available on the MCS web site 
as of the writing of this report. 


The first intervention component, Scholastic’s READ 180 program, has been implemented 
for four years in the MSRP schools. Students in the sixth through eighth grades who 
demonstrated the strongest need for reading support—i.e., performed in the bottom quartile on 
the reading/language arts portion of a prior Tennessee Comprehensive Assessment Program 
(TCAP)—were randomly selected to participate in the supplemental program. The READ 180 
intervention served 707 students in the eight schools in Year 1 (2006—2007), and 289 new sixth- 
grade students were assigned to the intervention in Year 2 (2007-2008). In Year 3, 274 new 
sixth-grade students were assigned to the READ /80 intervention, and in Year 4, 238 new sixth- 
grade students were assigned. 


The second intervention component, the Memphis Content Literacy Academy (MCLA), is a 
whole-school professional development program. The experimental research design in Years 1 
and 2 was a randomized matched-pairs design, with MCLA implemented in four of the eight 
schools. After the second year, MCLA ended in the experimental treatment schools and moved 
to the four schools that initially served as control schools. Operations in Years 3 and 4 thus 
represent the entirety of the whole-school intervention at the former control schools. The 
program, developed by University of Memphis and MCS staff, was originally designed for 
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teachers in the English/language arts (ELA), social studies, mathematics, or science content areas 
or special education teachers, but the program was expanded in Years 3 and 4 to include any full- 
time staff member who provided instruction to students (e.g., writing, foreign language, and 
exploratory teachers, guidance counselors, and instructional facilitators). In Year 4, a total of 
156 full-time school staff members were eligible to participate in MCLA. 


Background Context 


The United States Department of Education awarded MCS a five-year Striving Readers grant 
to help address the city’s significant educational needs. According to the MCS web site,” MCS 
serves more than 105,000 students and ranks as the nation’s twenty-third—largest K-12 school 
district, although total enrollment has dropped over the past several years. Over 95 percent of 
the 196 MCS schools are Title I schools, and 71 percent of students qualify for free or reduced- 
price meals (The Urban Child Institute, 2008). Approximately 86 percent of MCS students are 
African American, 8 percent are white, and 6 percent are other races and/or ethnicities (MCS, 
2009). The number of English Language Learners doubled between the 2000—2001 school year 
and the 2006-2007 school year, although numbers of students in this category remained 
relatively low among Striving Readers schools (MCS, 2009). Other data show that 71 percent of 
students in the sixth through eighth grades scored below the 50th percentile on the 
reading/language arts portion of TCAP (Potts, Perkins, Heeren, Harris, & Feldman, 2008). It 
appears that no progress has been made in raising students’ scores on the reading/language arts 
portion of the TCAP because the same percentage of students scored below proficient in 2008 as 
in 2005 (The Urban Child Institute, 2009). Research has shown that approximately 66.9 percent 
of MCS students graduate from high school within four years (The Urban Child Institute, 2009; 
Heart, 2008). 


Theoretical Rationale for and Description of the Targeted Intervention 
Model 


Description of the Targeted Intervention 


READ 180 is a commercially available reading intervention program from Scholastic that 
targets struggling readers in the fourth through twelfth grades. The Enterprise Edition is the 
most recent version, and it combines a software program, teacher-directed instruction using a 
textbook and similar resources, and independent or modeled reading (i.e., reading while listening 
to audiobooks). Close adherence to the structure of the program requires 90 minutes divided into 
four 20-minute and one 10-minute blocks. The first 20 minutes and a final 10-minute wrap-up 
involve whole-group instruction. The other three 20-minute blocks require students to rotate 
between teacher-led small-group instruction, individual use of the proprietary READ 180 
software, and reading leveled fiction and nonfiction texts provided with the program. 


During Years 1, through 4, there were two or three teachers teaching READ 180 at each of 
the eight MSRP schools (19 in each of Years | through 3; 16 in Year 4). Almost all of the 


* http://www.mesk12.net/facts_about_mcs.asp 
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teachers were licensed, experienced ELA teachers.’ Several schools offered READ 180 prior to 
the Striving Readers project, and in those instances, teachers with previous READ 180 
experience were assigned to teach READ 180 as part of the targeted intervention. There were no 
other criteria for assignment. 


Students Targeted by the Intervention 


MCS identified a pool of struggling readers at the eight MSRP schools and defined students 
who scored in the lowest quartile on the Reading/Language Arts section of the TCAP exam as 
eligible for assignment to READ 1/80. In fall of 2006, identified students were randomly 
assigned to the control or treatment condition, and in fall 2007, treatment students who were still 
enrolled in MSRP schools (that is, students who moved from sixth to seventh grade or seventh to 
eighth grade) were enrolled for a second year in READ 180.*Also, at the beginning of the 2007, 
2008, and 2009 school years, incoming sixth-grade students who were identified as struggling 
readers were randomly assigned to the control or experimental condition, and seventh-grade 
treatment students who remained in MSRP schools were again enrolled in READ /80 for a 
second year. Because READ /80 is considered a two-year intervention by MCS and this study’s 
design, eighth-grade students, whether they were originally assigned to the treatment or control 
condition, are not included in the Year 4 analyses. 


Students assigned to READ /80 experience the intervention in addition to their regular 
language arts classes and other courses related to language arts (e.g., reading, creative writing) 
that are offered at their schools. Control students experience the same language arts classes, 
reading classes, and/or other classes related to language arts that would be offered in their 
schools if MSRP did not exist. 


Logic Model for Targeted Intervention 


The logic model for the targeted intervention, as published in Scholastic’s READ 180 
Enterprise Edition Research Protocol and Tools (2007), appears as Figure IIH-2. Graphics 
illustrating the instructional model and detailing the rotation activities appear as Figures III- 
3aand III-3b. 


Professional Development Model Components 


In all four years, new READ /80 teachers were expected to attend two all-day training 
sessions (experienced teachers were expected to attend at least one). Additionally, the district 
hosted after-school, two-hour “networking meetings” (four during Years 1, 3, and 4, and seven 
during Year 2) at which Scholastic representatives and MCS staff members taught teachers how 
to use or improve different components of READ 180, such as using the data generated by the 
READ 180 software to differentiate instruction or using supplemental READ 180 teaching 
materials for strategic vocabulary instruction. During Year 2, MCS encouraged teachers to 
complete Scholastic’s “Best Practices for Reading Intervention,” a seven-part online course 


> One provisionally licensed teacher with one previous year of teaching experience taught READ 180 during Year 2 
* There was some attrition and a few instances in which students were opted out of the intervention; these are 
detailed in the “Sample Selection” section of Part IV. 
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designed for READ /80 classroom teachers (in Year 2, MCS provided a small monetary stipend 
to teachers who completed the online course). In Years 2 through 4, MCS staff members 
conducted observations of READ 180 teachers and provided feedback to help improve the 
teaching of READ 180; in Years 2 and 3, Scholastic representatives conducted at least one 
classroom observation per teacher and provided feedback to the teacher and to MCS district 
staff.> During Year 3, MCS began strongly encouraging school administrators to attend a half- 
day READ 180 training presented by Scholastic and designed for school leaders and 
administrators, and leaders were again encouraged to attend this training in Year 4, 


Figure II-1: Elements of READ 180 Professional Development, Years 1 through 4 


Element Year 1 Year 2 Year 3 Year 4 
All-day training* (seven hours) 2/1 2/1 2/1 2/1 
Networking sessions (two hours) 4 7 4 4 
Online course available incentivized not offered not offered 
Formative observation and at least 1 at least 1 at least 1 
not offered 
feedback per year per year per year 
Administrator training (four nebonered aetoneren 1 1 
hours) 


*New READ 180 teachers were required to attend two days of training; teachers who had experience teaching 
READ 180 were required to attend one day of training. 


Classroom Model Components 


The planned instructional model was to follow the published and recommended READ 180 
model. According to the Leadership Implementation Guide: Supporting READ 180 in Your 
District (2005) published by Scholastic, the recommended class size for READ 180 is 21 or 
fewer students (MCS strove for these classes to include 18 or fewer students). Scholastic 
recommends that students be divided into three homogenous groups according to diagnostic 
assessments and regrouped as indicated by subsequent assessments. (According to Scholastic’s 
READ 180 training materials, Scholastic allows for alternate grouping strategies, such as 
purposefully creating heterogeneous-ability groups or considering behavioral issues to guide 
grouping.) Students are to be in READ 180 class for 90 minutes during every school day. 
Figures III-3a and III-3b detail Scholastic’s recommendation for use of instructional time and 
provide some details about the targeted areas of reading and instructional approaches. 


READ 180 software provides instruction in decoding and word recognition, spelling, fluency, 
vocabulary, and comprehension. Whole-group and small-group instruction include a variety of 
instructional approaches, including fluency exercises, question stems, use of graphic organizers, 
activation of prior knowledge, and cooperative group work (among others). The Scholastic 
Achievement Manager (SAM) automatically generates student-level data based on work students 
have done and assessments they have completed using the READ 180 software. 


> Scholastic provides detailed descriptions of the all-day implementation training sessions and the online courses at, 
respectively, http://teacher.scholastic.com/products/read180/prof/implement_train.htm and 
http://teacher.scholastic.com/products/read180/prof/bestpractices.htm 
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All students in READ 180 classes are provided with a flexBook,° the course textbook. All 
READ 180 classrooms have libraries with a variety of fiction and nonfiction leveled texts 
provided for the modeled and independent reading rotation of READ 180. During the READ 180 
software rotation, each student has individual access to a computer to complete the exercises. 
Teachers use their dedicated computers for recordkeeping and for tracking student progress 
through use of the instructional software. 


READ 180 includes a number of assessment tools in its software. Regular reports of student 
progress through the instructional software are available to teachers through the networked 
computers using SAM and the Scholastic Management Suite (SMS) software. Teachers are 
expected to administer the Scholastic Reading Inventory (SRI) at least three times per school 
year; MCS established date “windows” within which the SRIs should be administered. 
Additionally, the teachers are expected to use the data from SAM and SMS, the SRIs, and other 
assessments deemed appropriate by the teacher to determine whether lessons are working, to 
differentiate instruction, and to regroup the students. 


° The rBook and the flexBook are two versions of the same text, with similar lessons, exercises, strategies, etc. The 
versions have different reading selections so students who are in READ 180 for two years do not experience 
identical readings over those two years. MCS alternates use of these texts each year. 
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Figure II-2: Logic Model of Targeted Intervention 
The READ 180 Enterprise Edition Logic Model 


Resources/Impacts 


Ongoing Teaching/ 
Learning Activities 


Short-Term Outcomes 


Long-Term Outcomes 


1) Teacher training and 
professional development” 


2) Scholastic Professional 
Development for 
administrators and 
technical directors 


3) rBook or flexBook 
Teacher’s Edition and 
rBook or flexBook student 
work text 


4) Networked computers with 
microphones and 
headsets, teacher 
workstation, and printer 

5) CD players with headsets 

6) TV with DVD player 

7) READ 180 EE Topic 
Software, Audiobooks, 
Paperbacks, and Anchor 
Videos 

8) Classroom space adequate 
for READ 180 instruction 

9) Scholastic Achievement 
Manager (SAM) 


A daily 90-minute instructional block 


20-minutes Whole-Group Instruction to 
Start the class 


Small-group rotations in which 
students are divided into 3 groups and 
spend 20 minutes each rotating 
through: 


*Small-Group Instruction 
*Modeled and Independent Reading 
*Use of READ 180 EE Topic Software 


Regular use by teachers of READ 180 
instructional strategies and materials 
contained in READ 180 program 
guides, which include (but are not 
limited to) independent reading of 
leveled texts, use of graphic 
organizers, and specific teaching of 
vocabulary. 

10 minutes of Whole-Group Wrap-Up to 
conclude the class 


Enrollment of 15-18 students per class 
[or up to 21] 


Enrollment lasting the entire school 
year [for two years] 


Instruction that follows rBook (or 
flexBook) scope and sequence 


Regular use, by teachers and 
administrators, of diagnostic tests (SRI) 
and the Scholastic Management Suite 
software for continuous assessment, 
placement, and monitoring 


Improved classroom 
behavior and school 
attendance and decreased 
disciplinary incidents 


Increased motivation and 
engagement in reading 


Improved state and local 
assessment results fat least 50% 
of the READ 180 program 
students will score proficient on 
TCAP reading, language arts, 
and other content area subtests, 
and those students will make a 
mean gain on reading and other 
content area subtests of at least 
5-10 NCEs over control students] 


Increased reading 
proficiency as reflected in 
SRI scores and other 
indicators monitored by 
SAM 


Improved learning in all content 
areas 


Contextual effects such as the characteristics of the school district, other 
instructional programs in use, and external events may also influence 


outcomes 


Logic Model copyright © 2007 Scholastic Inc. Text in blue italics is specific to MSRP. Blue underlined text was added for clarity by RBS. 
*The only differences in the targeted intervention between years 1 and 4 were related to professional development and are described in Section Ill. 
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Figure II-3a: READ 180 Instructional Model 


90-Minute Instructional Model! 
Small-Group Rotations w 


4. 
a 
[8 


“OD A ND INDEPENDES 


Figure II-3b: Description of READ 180 Rotation Activities 


e e e1K. 

Whole-Group Direct | Small-Group Direct READ 180 moe ae Whole-Group 
Instruction Instruction Software Reading Wrap-Up 
Using the Using the rBook Students use the Students build The session ends 

READ 180 and Resources for | software reading with 10 more 
instructional Differentiated independently, comprehension minutes of whole- 


materials, the 
teacher begins the 
day by providing 
systematic 
instruction in 
reading, writing, 
and vocabulary to 
the whole class. 


Instruction, the 
teacher works 
closely with 
students so that 
individual needs 
can be met. 


providing them with 
intensive, 
individualized skills 
practice. 


skills through 
modeled and 
independent 
reading of the 
READ 180 
paperbacks and 
audiobooks. 


group instruction. 


The above graphic and table were copied on November 8, 2007, from 
http://teacher.scholastic.com/products/read 1 80/overview/instrmodel.htm#small-group 
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Theoretical Rationale for and Description of Whole-School Intervention 
Model 


MCLA is a whole-school intervention designed to improve teaching and learning through 
intensive professional development for teachers, onsite literacy coaching assistance, a seminar 
for principals and other school leaders, and classroom use of grant-funded curricular resources. 
In the first two years of the Striving Readers grant, four of the eight participating middle schools 
(hereafter referred to as Cohort | school”) were randomly assigned to receive the whole-school 
intervention, while four schools serving as research controls (hereafter designated as the Cohort 
2 school’) did not participate in MCLA. In Year 3, the MCLA intervention moved from the 
Cohort | to the Cohort 2 schools, and developers invited all Cohort 2 full-time staff members 
with teaching responsibilities to participate in two years of professional development that would 
focus on infusing literacy into the core academic content areas English/Language Arts (ELA), 
mathematics, science, and social studies. Special education and” exploratory” teachers (e.g., 
those teaching art, music, or computers), school counselors, librarians, and others who provide 
instruction to groups of students were also encouraged to join MCLA in Years 3 and 4. A total 
of 144 full-time school staff members at the four Cohort 2 schools were eligible to participate in 
MCLA in fall 2008, and 156 were eligible in fall 2009. 


Developers hypothesized that greater and more effective integration of literacy strategies by 
teachers would lead to student performance improvements in reading and the core academic 
content areas. The intervention was designed so that teachers with no prior knowledge of or 
experience with literacy integration had opportunities to practice strategies in the MCLA course 
and then, with coaching support, gradually assume responsibility for helping their students learn 
how and when to use the techniques on their own. Two tools designed for the evaluation 
elaborate on this theory of action: the first (Figure 4) is a logic model of intervention activities, 
outputs, and anticipated outcomes; the second (included in Appendix A) is a significantly more 
comprehensive and in-depth rendering of the intervention, an “Innovation Configuration (IC) 
Map,” created by a partnership of evaluators, program developers, and project implementation 
leaders and staff. 


As Figure 4 shows, developers originally planned to offer content-related materials on 
literacy integration in the MCLA evening course and, in each school, a $40,000 curriculum 
resource center (CRC) stocked with content-rich, multi-leveled materials and an onsite literacy 
coach to assist and support teachers with strategy implementation. Developers anticipated that 
teachers would, as a result of participation, gain a deeper understanding of the need for literacy 
integration into the content areas and use a series of research-based literacy integration skills 
with increased frequency and confidence. 


Developers also hoped that the intervention would create a sharpened schoolwide focus on 
adolescent literacy as teachers shared techniques and experiences and assessed the program’s 
impact on their students. Ultimately, developers anticipated that the transformed literacy- 
saturated middle school environment would boost student achievement first in reading and then 
in the core content-area classes. 
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FigurellI-4: Logic Model of the Memphis Striving Readers Whole-School Intervention 
OUTCOMES 


INPUTS 


ACTIVITIES/OUTPUTS 


Teacher professional 
development—Memphis 
Content Literacy Academy 
(MCLA) 


30 weekly 3-hour sessions 
for a total of 180 hours over 
two years designed to train 
teachers to develop and 
implement eight classroom 
action plans each year 


Provide coaching on site for 
core content teachers 


Principal professional 
development—Fellowship 


45-hour course, over a two- 
year period, trains principals 
to 

Provide teachers with 
feedback from classroom 
walkthroughs 

Explore schoolwide factors 
principals can influence to 
sustain MCLA 


Incorporate literacy into the 
school improvement plans 


MCLA Coach professional 
development 


MCLA Instructor 
professional development 


Core content teachers plan 
and implement lessons 
integrating literacy strategies 
within their content areas: 


* Use assessments of 
student knowledge and 
literacy abilities to plan 
instruction to meet the 
needs of all learners 


Provide explicit and direct 
instruction and practice 
incorporating appropriate 
literacy strategies matched 
to content learning 
objectives 


Use of set content area 
standards for their 
instruction plans and 
identify literacy strategies 
students will use with 
relevant texts 


Use supplementary 
content-relevant reading 
materials to meet 
individual student needs 


Design and use 
cooperative learning 
activities to provide 
students extensive practice 
opportunities 


Collaborate with other core 
content teachers to work 
on integrating literacy 
strategies in core content 
lessons 


| ad 


Students use 
literacy strategies 
when reading 
content-relevant 
texts for core 
content classes, 
specifically: 
students use 
before-, during-, and 
after-reading 
strategies to 
understand and 
learn from grade- 
level content-related 
texts. These 
strategies help them 
develop 


° Fluency 
* Vocabulary 


* Comprehension 


Student assumes 
appropriate roles 
and responsibilities 
during cooperative 
learning activities 


Improved student 
performance on 
TCAP and ITBS 


I 


i 


i 


Schoolwide Factors 


(1) Principal leadership, (2) school culture supportive of the use of literacy strategies in core content classes, 
(3) environment press by number/percentage of core content teachers that have been trained by MCLA and who 


are integrating literacy strategies in their content lessons 


Notes: The logic model describes the two-year intervention as planned. Details about the intervention as 
implemented in Year 4 are presented in section V (e.g., developers provided 42.5 hours of MCLA course-related 
professional development in Year 4) 


The Innovation Configuration Map 


During Years 1, 2, and 3 of MSRP, the team of university, school district, and evaluation 
partners developed an IC Map (Hall and Hord, 2006) that explicated the intervention’s main 
components. The IC Map provides operational definitions of program components at various 
levels of implementation and can be used to reflect on teacher practice, monitor classroom 
implementation, identify the areas of a program in most need of attention, and develop 
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evaluation instruments. It was finalized and piloted with literacy coaches in Year 3 and 
administered in Year 4; ratings from the IC Map are currently being correlated with student 
outcomes to empirically establish which MCLA components, if any, are most strongly associated 
with targeted improvements. 


More specifically, in spring 2010 RBS evaluators engaged the MCLA literacy coaches in 
rating teachers’ level of implementation fidelity with respect to specific classroom practices. 
The teacher-related practices rated by coaches during a one-and-a-half-day retreat designed and 
facilitated by RBS included the following: 


* introducing literacy strategies to students 

* modeling how to use the strategies 

* providing instruction that is explicit and direct 

* differentiating instruction 

* enabling students to use strategies independently 
* revisiting strategies consistently during lessons 


Throughout the school year, coaches collected evidence about teacher implementation such 
as observation notes, student work products, and class handouts. Before using the IC Map to rate 
fidelity implementing the specific components, evaluators asked the coaches to assign global 
ratings (i.e., low, medium, high) for individual teachers based on their overall impression of the 
teachers’ highest levels of implementing a composite comprised of all five components. 
Coaches drew upon their professional experience with teachers and data sources in each 
teacher’s portfolio and issued a separate rating for each component. A cluster analysis 
conducted by RBS showed significant agreement between coaches’ initial global ratings 
assigned to teachers and computer-generated clusters based on specific ratings of each 
component. Clusters indicated that the IC Map was useful in describing observable aspects of 
teaching and learning and sufficiently sensitive to distinguish between various levels of 
implementation fidelity. (Additional details and findings from the Year 4 implementation 
fidelity rating activity involving coaches are presented in section IV of this report.) 


Professional Development Model Components 


The MCLA teacher and principal course syllabi, coach job descriptions, and an inventory of 
instructional materials together provide the best description of the intervention’s four main 
components. The teacher course was designed to meet weekly for three hours over two years, 
for a total of 180 hours of professional development. According to the original proposal, 
planners had hoped to enroll 80 percent of approximately 120 eligible content-area teachers who 
would select one of four evening content courses (mathematics, science, ELA, and social studies) 
according to their primary teaching assignment. Table II-2 provides details about the PD model 
as planned and as implemented in Years 1 through 4. 


To ensure consistency across classes, developers prepared templates for instructors who were 
highly qualified reading and content-area specialists to follow when teaching the course; these 
templates were similar structurally across the content areas. Developers also provided 
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discipline-specific materials for teachers. The developers’ goal was for each participant to see 
strategies modeled during the evening course and then, with a coach’s guidance, practice using 
those strategies in his or her own classroom. By issuing approximately ten “classroom action 
plan” (CAP) course assignments over two semesters, designers hoped that teachers would be 
compelled to work with coaches in school on a weekly basis. 


Developers instituted monthly Principal Fellowship meetings to apprise school principals of 
MCLA activities and to provide them with opportunities to brainstorm possible solutions to 
commonly experienced problems. In Year 4, developers invited building principals, assistant 
principals, and instructional facilitators as a team to attend four sessions between September 
2009 and February 2010; these teams were created and invited in an effort to mitigate erosion of 
support resulting from principal turnover and/or ongoing shifts in district-level priorities. 


Assistance provided by onsite literacy coaches is the third component of MCLA. In Year 4, 
the six literacy coaches—individuals with at least five years of teaching experience, a Master’s 
degree, and strong literacy background—were responsible for observing, monitoring, and 
assisting teachers with meeting the weekly objectives of the CAPs; monitoring READ 180 
program delivery; visiting with principals to keep them informed of teachers’ needs; and 
maintaining the CRC. The coaches were also expected to participate in a wide range of 
professional development activities provided by MCLA developers in the areas of mentorship, 
urban education, adolescent literacy, and in professional development related to READ 180. 


The CRC is the fourth and final component of the MCLA professional development program. 
Housed in each of the participating schools, each CRC contains leveled books, kits, reference 
books, and other materials to assist teachers with integrating literacy into their content area 
classes. Evening course instructors and onsite literacy coaches promoted the use of CRC 
materials, which remained in the schools when MCLA ended. 


Classroom Instruction Model Components 


Prior to implementation, MCLA designers identified twelve key instructional strategies as the 
primary foci of the evening course, including the use of graphic organizers, comprehension 
monitoring techniques, question generation, repeated oral reading, preteaching vocabulary, and 
direct, explicit instruction. Year 3 and Year 4 course assignments required teachers to use 
strategies aimed at improving students’ vocabulary, fluency, and comprehension and to meet 
with coaches to plan lessons designed to bolster strategy implementation. The MCLA course 
instructional model was designed to provide opportunities for teachers to practice modeling the 
literacy strategies among colleagues as part of training before implementing them with their 
students. Participants were then expected to model the strategies for their students and help the 
students adopt and use the strategies with increasing independence. Developers had hoped that 
teachers would integrate the strategies into existing class activities rather than view them as 
separate and distinctive lessons. Although developers did not plan a formal system for using 
student data to inform decision-making in MCLA, they built into the CAP assignments an 
informal student assessment procedure. 
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Changes Made to the Professional Development Model between Cohorts 1 and 2 


MCLA was administered to Cohort 1 teachers during the first two years, while Cohort 2 
teachers served as a control group. In Years 3 and 4, developers provided staff development and 
support to Cohort 2 teachers (the program ended in Cohort | schools). During the first two years 
of MCLA, developers fine-tuned service delivery based on experience gained from 
implementation, the feedback of participants, and data collected by evaluators. Before rolling 
out the Year 3 intervention, developers and school district staff made the following changes: (1) 
allowing all school staff who provided instruction to students to participate in MCLA and (2) 
permitting nonparticipants to borrow resources from the CRC. 


Activities in the teacher course during Years 3 and 4 continued to be organized into three- 
week cycles: the first week focused on an explication of key strategies, the second week allowed 
teachers to model the strategies through presentations, and the third week offered them 


opportunities to work collaboratively with colleagues to develop their classroom action plans 
(CAPs). 


TablelI-2: Changes to MCLA Professional Development Model, Years 1 through 4 


Total Hours Teacher Total Hours 
Changes to PD Model Course PD Principal PD 
90 
Planned per year for two years 45 
(total 180 hours) over two years 
Actual 
Year 1 n/a 85.0 20 
Assistant principals invited to attend 
the principal training. Special 
education teachers invited to attend 18 
Year 2 the teacher trainings. Course format 56.5 (six 3-hour 
changed to rely more heavily on sessions) 
small-group activities and class 
presentations. 
Teacher course offered to all staff who 21 
Year 3 provide instruction. Nonparticipants 49.0 (seven 3-hour 
permitted to use CRC materials. sessions) 
Teacher course offered to all staff who 12 
Year 4 provide instruction. Nonparticipants 42.5 (four 3-hour 
permitted to use CRC materials. sessions) 


Data sources: University of Memphis and MCS records 


Brief Overview of Key Evaluation Design Features 


The evaluation design for the MSRP study addresses the impacts of READ 180 and MCLA 
on student and (in the case of MCLA) teacher outcomes. The measures of student outcomes are 
the Iowa Tests of Basic Skills (ITBS) to measure reading achievement, and the TCAP, which 
measures achievement in reading/language arts, mathematics, science, and social studies. 
Outcome measures from the ITBS include total reading standard, vocabulary, and 
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comprehension scores. Outcome measures from the TCAP include scale scores in the four 


content areas. 


There are two teacher outcome measures: an index of the teacher’s perceived preparation to 
employ literacy strategies in the classroom and an index of the teacher’s perception of how 
frequently these strategies are employed. Each index is based on ratings of preparation or 


frequency for 24 literacy strategies. 


Summary of Year 4 RBS Data-Collection Activities 


Figure II-5 describes the methods RBS used to collect information for the Year 4 MSRP 
evaluation, the time frame in which data were collected, the sample sizes obtained, and 
associated response rates (indicated parenthetically in the sample size column). As shown, RBS 
collected information from participants through surveys, interviews, classroom observations, 
documentation review (i.e., analysis of program logs), and student assessments. Information was 
collected in the fall and spring semesters, and response rates were high for most sources of data. 


Figure II-5: Characteristics of Year 4 RBS Data-Collection Methods 


Data-Collection Method and Topic 


Date Conducted 


Sample size* 


Feedback Survey (MCLA) 


N = 214 (98.1%)! 


N = 93 (100%) completed at least 1 survey 
N =77 (82.8%) completed surveys at 
baseline, mid-term, and at follow-up. 


Interviews 


.. Striving Readers School Principals | May 2010 
Literacy coaches May 2010 
| Observations: 
October 2010 
READ 180 classrooms 
ee een ete May 2010 
. READ 180 classrooms | April/May 2010 | 
October 2009 
MCLA Classrooms 
May 2010 


Student assessment 


Follow-up ITBS 


Spring 2010 


Secondary Data 


MCLA attendance rosters, 
READ 180meeting attendance 
sheets 


TCAP 


Spring 2010 


*Where possible, response rates are provided in parentheses. 

Teacher Implementation of Strategies Questionnaire 

? Response rate was calculated using MCS data file that lists all MSRP content, exploratory, and special education 
teachers (spring 2010). TotalN = 233 teachers; this is the denominator used to calculate percentages. 


All available data 


Not known at this time 
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lll. Evaluation of the Implementation of the Targeted 
Intervention, Years 1 through 4 


Summary of the Design 


Evaluators from RBS conducted many activities examining implementation of READ 180 in 
MCS Striving Readers schools during Year 1 (the 2006-2007 school year); however, the 
requirement for creating a more in-depth analysis of implementation was not instituted until Year 
2 (the 2007—2008 school year). In response to this requirement, a plan to study implementation 
more closely and develop classroom implementation ratings was created during Year 2. The 
efforts and procedures for this close study and the development of implementation ratings 
continued through Year 4 (2009-2010). The research questions created to evaluate the 
implementation of the targeted intervention are as follows: 


1. What were the levels and variability of implementation of teacher professional 
development in Years | through 4? 


2. What were the levels and variability of implementation at the classroom level in Years 1 
through 4? 


Table ITI-1 lists the research questions and indicates the relevant data that are available from 
the four years of implementation. As in Year 1, data from attendance records maintained by 
MCS were used to inform evaluation of professional development (PD) implementation, and 
Scholastic Achievement Manager (SAM) and observation data were used in the calculation of 
classroom implementation ratings. In Year 4, four rounds of classroom observations were 
conducted: twice by the evaluator, and twice by MCS. This is fewer rounds of observation than 
in Years 2 and 3 for two reasons: one data-gathering trip scheduled by the evaluator was 
cancelled due severe weather, and MCS decided that the observation feedback provided by 
Scholastic (as in Year 3) was not helpful. MCS did not survey students or teachers from READ 
180 classrooms in Year 4; therefore, no survey data are available for use in analyzing 
implementation. Open-text responses from classroom observations and brief interviews with 
teachers conducted immediately after evaluator observations were reviewed and used to 
triangulate quantitative data from the implementation ratings. 


Development of the Ratings and Scale for Years I through 4 


As noted in Table II-1, the sources of data for rating the implementation fidelity of 
READ 180 included classroom observations (copies of the observation instruments can be found 
in Appendix B), data generated by SAM, and district documentation related to professional 
development. Findings from all of these sources were translated to a 4-point scale ranging from 
1 to 4.’ For all ratings, “adequate” is defined as 3 or above—the “moderate” or “high” level (on 
this scale, 2 is defined as “low,” and | is defined as ““minimal’’). The “Professional Development 


’ The Year 1 Executive Summary of Implementation submitted in August 2008 presented classroom ratings on a 
four-point scale that ranged from 0 to 3. These numbers have been changed to match the scales of Years 2 
through 4 and of the whole-school implementation ratings. 
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Scales” and “Levels and Variability of Implementation at the Classroom Level” sections each 
include more detailed descriptions of the specific data sources used to rate each of those areas. 


Table III-1: Years 1 through 4 Data Sources Linked with Implementation Research 
Questions—Targeted Intervention 


Research Questions Measures/Data Sources 
Pot—“*‘CS*S*S*S*S~SCSCSCS*dEC Surveys | SAM [Observations | Record Review | 


MCS—PD Attendance 


& online course 
documentation and 


Developer/District 
Evaluator 
Developer 

District staff 
completion records 
Developer 
handouts 


What were the levels and variability of implementation of teacher professional development in 
Years 1 and 2? 


Types/amount of professional 
development provided to 
teachers 


NO 


Proportion of teachers at 
different levels of professional 
development 


Proportion of teachers at 
adequate level of professional 
development 


Types/amount of professional 
development provided to district 
leaders 


Proportion of leaders at different 
levels of professional 
development 


What were the levels and variability of implementation at the classroom level in Years 1 and 2? 


sipped witimeterale v1 | YES: Yrs. | Yrs. | Yrs. 
PP 283 14 | 283 | 24 
resources, and technology 

Classrooms in which model was vr 4 Yrs. Yrs. Yrs. Yrs. Yrs. 
implemented at different levels , 283 1-4 1-4 283 2-4 
implemented aradequatonever | Aga | 06 [ev tSer| Me Yass | re 
e q 283. | 14 | 14 | 283 | 24 
or above 


Note: Teachers do not always control whether a class they are teaching is on model. For 
example, if aREAD 180 computer server is down or students are taking a state assessment, the 
average number of sessions per week that students log on to the computers will drop. Therefore, 
it is important not to consider these simply “teacher” ratings. Also, students can be linked with 
their teachers; however, students of any one teacher cannot be further divided into the class 
sections that the teacher leads. Therefore, the ratings are most appropriately considered to be at 
the classroom level rather than the teacher or class period level. 
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Year 4 Implementation Study 


Examination of implementation fidelity in Year 4 followed the same procedures established 
for the report of fidelity in Years 2 and 3. In fall 2008 (the beginning of the Year 3 school year), 
as part of its efforts to serve students who were expected (by the district) to benefit from extra 
instruction to improve their reading skills, MCS began offering more READ /80 classes in more 
middle and high schools. This effort substantially increased the number of schools that offered 
READ 180 and reflected a districtwide emphasis on the program. The emphasis was not specific 
to Striving Readers schools; however, they were affected and involved. This increase in 
emphasis on READ /80 program components during Year 3 largely continued during Year 4 and 
may be related to the increase in the amount of time students spent using READ 180 computer 
software (see Table III-6). 


Professional Development Levels 


In Year 4, the professional development participation score again was developed by 
combining the numbers of points assigned to different types of professional development. Data 
related to all 16 Year 4 READ 180 teachers were drawn from MCS PD sign-in sheets and other 
documentation provided by project staff members. During the 2008—2009 school year, MCS 
began emphasizing training for administrators in schools that offered READ 180. Therefore, in 
addition to the same four types of professional development recognized in Year 2, the PD ratings 
for Year 3 included an extra point if an administrator (or someone serving in a coaching or PD- 
facilitating role) attended a half-day administrator training; this “administrator point” was also 
included in the Year 4 calculations. MCS did not emphasize the Scholastic online course in 
Years 3 or 4; however, many teachers completed the course in Year 2, and three points were 
added to the calculations of Year 4 ratings of teachers who had completed the online course. 
Similarly, READ 180 teachers who attended networking meetings and/or full-day trainings in 
Year 3 were awarded additional points toward the Year 4 ratings. According to district staff 
members, the Year 4 sessions and meetings included some information new in Year 4 and some 
that had been presented during Year 3. Therefore, teachers were awarded half of the number of 
points for attendance at Year 3 compared to Year 4 trainings. Further theorizing that teachers 
who are more familiar with the materials and procedures of READ /80 are better prepared to 
teach it, RBS evaluators awarded teachers who had taught READ /80 in previous years 
additional points, as noted below. 


attendance at any all-day session earned 2 points for each session attended (attendance at 
each all-day session in 2008—2009 earned 1) 


attendance at any networking meeting earned 1| point for each meeting attended (attendance 
at each networking meeting in 2008—2009 earned 0.5) 


the first year of experience teaching READ 180 earned 2 points, and any number of years of 
experience beyond that earned 3 


completion of the Scholastic’s online course earned 3 points 


attendance at administrator training by an administrator or literacy coach earned each of 
that school’s teachers | point 
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This resulted in a possible total of 19 points. Evaluators used the same equation of raw score 
to ratings in Years 3 and 4 as in Year 2: professional development scale scores greater than or 
equal to ten are considered “high,” those seven or higher are considered “moderate,” those four 
or higher are “low,” and scores three and below are “minimal.” These ratings are included in the 
“Rating” column of Table II-2. In order to receive a “high” rating, a teacher would have to 
participate in at least three types of professional development or at least two types if someone 
from his or her school attended administrator training. Table II-2 indicates the number and type 
of professional development opportunities completed by all 16 Year 4 teachers. 


Table IlI-2: Teacher Completion of Year 4 READ 180 Professional Development 
Opportunities (N = 16) 


Full Network Online 
Unique ID Days Mtgs. Yrs. Exp. | course Admin. Score Rating 
8348 2 3 3 Y Y 16 4 
2109 2 4 3 N Y 15 4 
5224 2 1 3 Y N 13 4 
5102 3 0 3 Y Y 13 4 
4781 2 0 3 Y N 12 4 
5541 2 0 3 Y Y 11 4 
6033 2 1 3 N Y 11 4 
4420 1 0 3 Y Y 10.5 4 
5546 1 1 3 Y N 10 4 
3973 1 2 3 N Y 9.5 4 
4536 2 0 2 N Y 9 3 
6410 2 0 3 N Y 9 3 
3328 1 0 3 N Y 7.5 3 
3771 2 0 2 N Y 7 3 
8177 2 1 0 N Y 6 2 
8933 1 0 0 N N 2 1 


Data source: Training sign-in sheets provided by MCS 
Notes: Ratings defined as “adequate” or above are in bold. [D numbers were randomly generated and are used 
only to identify classrooms from year to year. All ratings are on a 4-point scale in which 4= high, 3=moderate, 

2=low, and 1=minimal; “adequate” is defined as 3 or above. 


Professional Development Levels in all Four Years 


Table III-3 lists the professional development ratings for all teachers in Striving Readers 
schools in all four years of the project. As shown in the table, the average PD ratings increased 
steadily over the four years of the project. 
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Table IlI-3: Teacher Professional Development Ratings, Years 1 through 4 


Unique ID Year 1 Year 2 Year 3 Year 4 Average 
2026 -- 4 -- -- 4.0 
2109 4 4 4 4 4.0 
5102 -- -- -- 4 4.0 
6033 3 4 4 4 3.8 
5546 - 3 4 4 3.7 
4420 4 2 4 4 3.5 
4781 4 2 4 4 3.5 
6132 3 4 - - 3.5 
8348 3 3 4 4 3.5 
3566 3 3 4 3.3 
5224 3 2 4 4 3.3 
1223 3 - - - 3.0 
1988 =- on 3 - 3.0 
3771 - - - 3 3.0 
3801 - 3 - - 3.0 
4536 - - 3 3 3.0 
5133 3 - - - 3.0 
5163 3 - - - 3.0 
5499 - 3 - - 3.0 
5515 - 3 - - 3.0 
5541 3 3 2 4 3.0 
6135 - - 3 - 3.0 
9631 - - 3 - 3.0 
3973 1 3 3 4 2.8 
3328 3 3 2.7 
5535 3 - 2 - 2.5 
6410 1 3 3 3 2.5 
2918 2 3 2 - 2.3 
2983 2 - - - 2.0 
8177 -- -- -- 2 2.0 
8213 2 - - - 2.0 
8877 - 2 - - 2.0 
6684 -- -- 1 -- 1.0 
7273 1 - - - 1.0 
8933 - -- - 1 1.0 

Average 2.7 2.9 3.2 3.4 


Data sources: MSRP Implementation reports, Years 1 through 4 


Levels and Variability of Implementation at the Classroom Level 


Implementation fidelity and variability were informed by data collected through classroom 
observations and data generated by the Scholastic Achievement Manager (SAM). In Year 4, as 
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in Years | through 3, the focus for assessment of implementation was the resources and 
instruction that were made available to or completed by students, not how well students 
performed on tasks or assessments. 


SAM captures data as students use the READ 180 software during the computer rotation. 
Teachers also have the option of entering additional data, and as part of the model, teachers are 
encouraged to generate automated reports about student progress from SAM on a regular basis 
(the actual time between reports varies from weekly to about quarterly depending on the report). 
Table III-4 presents the frequencies with which teachers say they used SAM reports. 


Table III-4: Reported Frequencies with which READ 180 Teachers Used SAM Reports 
(N = 16) 


Report name Number 
More than weekly 10 
Weekly 1 
Quarterly 2 
Missing* 3 


Data source: Postobservation interviews of READ 180 teachers conducted by RBS in spring 2010 
*Two teachers were not available to be interviewed after the observation, one teacher refused to be observed or 
interviewed. 


As part of creating the classroom ratings, RBS evaluators included two SAM variables 
related to student use of the software: the average number of sessions per week that students used 
the software and the median number of minutes students spent using the software during each 
daily session. Perfect implementation of READ /80 involves students working in the computer 
rotation for 20 minutes every school day. Recognizing that students need time to move from 
rotation to rotation during class and that school schedules often fluctuate, Scholastic has 
identified adequate implementation at 15 minutes per day at least 3 days per week. Figures IL-1 
and III-2 provide cross-year comparisons of the median number of minutes per session and the 
average number of sessions per week. As can be seen in Figure III-1, by Year 4, all schools had 
at least 15 minutes per student per day using READ 180 software; the average across schools is 
17.9 minutes. Similarly, as shown in Figure III-2, the average number of days per week 
increased from 2.3 to 2.7, and the lowest number of days per week increased from 1.7 to 2.2. 


The data represented in Figures IH-1 and III-2 show school-level differences in SAM data. 
To determine whether there are significant classroom-level differences between Years 3 and 4, 
evaluators conducted independent t-tests. These tests indicate that there was a significant 
improvement from 2.217 sessions per week in Year 3 to 2.752 sessions per week in Year 4 
[¢(33) = 5.161, p = .000], but there was no significant difference in the median amount of time 
per session. 
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Figure III-1: Year-by-Year Averages of the Median Amount of Time (in minutes) that 
Students Spent Using READ 180 Software per Session in each School 
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Data source: SAM (from years 1 through 4) 


However, when evaluators analyzed data from the students of the 12 teachers who taught in 
both Years 3 and 4, paired-samples t-tests showed significant differences in both average 
sessions per week and median minutes per session. The number of sessions per week increased 
from 2.242 in Year 3 to 2.772 in Year 4 [¢(11) = 6.932, p = .000], and the median number of 
minutes per session increased from 16.19 in Year 3 to 18.48 in Year 4 [¢#(11)= 3.651, p = .004]. 
While the number of sessions per week remains below the Scholastic-recommended number (3), 
the number of minutes in each session is substantially higher than the recommendation of 15. 


By multiplying the average number of sessions per week by the median number of minutes 
per session and by the number of weeks in a school year (36), researchers found that on average, 
students in Year 4 classes of teachers who had also taught in Year 3 experienced 545 more 
minutes using the READ /80 software than their Year 3 counterparts [¢(11) = 5.622, p = .000]. 
These findings seem to indicate that the district has decreased the amount of time that computer 
problems prevented use of the software and/or that, as teachers gain experience teaching 
READ 180, they have increased the amount of time their students spend using the software. 


As in Year 3, evaluators examined all included variables and created an equation for 
translating each SAM or observational variable to the four-point scale. Next, an equation was 
created that encompassed the data from all observations into one overall observation rating for 
each classroom (this equation appears as Appendix C). Finally, the overall observation rating, 
the SAM rating, and the PD rating were averaged into one overall classroom rating. 


All of the 16 classrooms had SAM data; each classroom was observed at least twice. Whole- 
group instruction and independent reading are activities that, according to interviews with district 
staff members, occur in many regular language arts classes. Therefore, data related to the small- 
group instruction and computer rotations were weighted more heavily because they are 
components of the READ 180 program that distinguish the program from the regular language 
arts classes offered. Also, in the calculations of overall classroom implementation level, 
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observations were weighted the most heavily because they addressed all components of 
READ 180. SAM data and the professional development scale followed these in weight. 


Figure III-2: Year-by-Year Averages of the Number of READ 180 Software-Rotation 
Sessions Students Completed each Week in each School 
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Data source: SAM (from years 1 through 4) 


During the 2009-2010 school year, members of the evaluation team completed classroom 
observations (NV = 30) during October (n = 15) and March (nm = 15). MCS staff members 
completed observations (NV = 31) during October (n = 15) and during February and March 
(n = 16). Observers from RBS used different observation protocols than MCS observers. 
Therefore, evaluators identified items that were similar across the two protocols and used those 
to calculate the observation ratings. The items used related to the following: 


the timing of the class (that is, whether the class had four 20-minute and one 10-minute 
segments) 


the number of students (seven or fewer) in each small group 


the layout of the room (structured to support the requirements of all rotations) 


content and student engagement in whole-group instruction and the small-group 
instruction, computer, and independent reading rotations 


use of the Scholastic flexBook and/or other READ /80 materials for instruction 


To create ratings for the computer rotation, RBS evaluators linked students to their 
READ 180 teachers and averaged the available data from all students taught by each teacher. As 
noted above, the SAM variables used for this report included the following: 


* average number of READ /80 computer software sessions per week 


* median daily number of minutes spent in these sessions 


Table III-5 presents the completed Year 4 ratings for all READ /80 teachers/classrooms in 
Striving Readers Schools. The ratings on professional development, from observations, and 
from SAM were averaged to create the overall rating for each teacher. 
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Table IlI-5: Year 4READ 180 Fidelity of Implementation Teacher/Classroom 


Ratings (N = 16) 


Professional 
Classroom ID Development Observations SAM Overall Rating 
8348 4 3.7 4.0 3.9 
6033 4 3.9 3.0 3.6 
4781 4 3.7 3.0 3.6 
5224 4 3.6 3.0 3.5 
3328 3 3.4 4.0 3.5 
4420 4 3.3 3.0 3.4 
6410 3 3.2 4.0 3.4 
5541 4 3.9 2.0 3.3 
3973 4 2.9 3.0 3.3 
3771 3 3.8 3.0 3.3 
5102 4 3.7 2.0 3.2 
2109 4 2.4 3.0 3.1 
5546 4 3.3 2.0 3.1 
4536 3 3.7 2.0 2.9 
8177 2 3.8 2.0 2.6 
8933 1 3.3 3.0 2.4 
Number “Adequate” 14 14 11 13 
Percent “Adequate” 87.50 87.50 68.75 81.25 


Notes: ID numbers were randomly generated and are used only to identify classrooms from year to year. All ratings 
are on a 4-point scale in which 4 = high, 3= moderate, 2 =low, and 1= minimal; “adequate” is defined as 3 or 


above. 


Conclusions Regarding Implementation of the Targeted Intervention 


Analyses of data related to READ 180 implementation indicate that there is still a fairly wide 
range of implementation across classrooms in the eight Striving Readers schools and that 
implementation has improved each year. Figure III-3 shows the percentages of classrooms that 
were rated as adequate in Years 1 through 4. 


Direct comparisons within 12 classrooms are possible, and those comparisons are presented 
in Table III-6 (because of teacher turnover, direct comparisons in the remaining classes are not 
possible). As shown in the “Overall Rating” column in Table III-6, nine of twelve classrooms 
(75%) had improved classroom ratings in Year 4 over Year 3, and there was an average increase 


in every area. 


Memphis Striving Readers/RBS: Evaluation and Impact Report 


June 17, 2011—Page 26 of 114 


Figure III-3: Percentages of Classrooms Rated Adequate in Years 1 Through 4 
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Note: Readers are cautioned against inferring too much from the comparisons between Years 1 and 2 because of 
differences in PD implementation and in the amounts of data available. N = 19 in Years 1, 2, 3; N = 16 in Year 4. 


Table IlI-6: Differences Between Ratings of Classrooms from 2008-2009 to 2009-2010 

(N = 12) 

Classroom Professional 
ID Development Observation Ratings SAM Overall Rating 


Y3 Y4 A Y3 Y4 A Y3 Y4 A Y3 Y4 A 


2109 4.00 | 4.00 | 0.00 | 3.40 | 2.42 | -0.98 | 2.50 | 3.00 | 0.50 | 3.28 | 3.14 | -0.14 


3328 3.00 | 3.00 | 0.00 | 3.10 | 3.39 | 0.29 | 3.00 | 4.00 | 1.00 | 3.13 | 3.46 | 0.34 


3973 3.00 | 4.00 | 1.00 | 2.90 | 2.90 | 0.00 | 3.00 | 3.00 | 0.00 | 3.15 | 3.30 | 0.15 


4420 4.00 | 4.00 | 0.00 | 3.20 | 3.33 | 0.13 | 3.50 | 3.00 | -0.50 | 3.33 | 3.45 | 0.12 


4536 3.00 | 3.00 | 0.00 | 3.20 | 3.67 | 0.47 | 2.50 | 2.00 | -0.50 | 3.10 | 2.89 | -0.21 


4781 4.00 | 4.00 | 0.00 | 3.40 | 3.72 | 0.32 | 3.00 | 3.00 | 0.00 | 3.33 | 3.57 | 0.25 


5224 4.00 | 4.00 | 0.00 | 3.10 | 3.57 | 0.47 | 2.50 | 3.00 | 0.50 | 3.25 | 3.52 | 0.27 


5541 2.00 | 4.00 | 2.00 | 3.40 | 3.92 | 0.52 | 2.50 | 2.00 | -0.50 | 2.75 | 3.31 | 0.56 


5546 4.00 | 4.00 | 0.00 | 3.10 | 3.29 | 0.19 | 2.50 | 2.00 | -0.50 | 3.18 | 3.09 | -0.08 


6033 4.00 | 4.00 | 0.00 | 3.30 | 3.94 | 0.64 | 3.00 | 3.00 | 0.00 | 3.43 | 3.65 | 0.22 


6410 3.00 | 3.00 | 0.00 | 3.50 | 3.15 | -0.35 | 3.00 | 4.00 | 1.00 | 3.20 | 3.38 | 0.18 


8348 4.00 | 4.00 | 0.00 | 3.60 | 3.73 | 0.13 | 3.00 | 4.00 | 1.00 | 3.48 | 3.91 | 0.44 
Average A, Y3 to Y4 0.25 0.15 0.17 0.18 


Data source: ratings developed and presented in this report and the Year 3 report 


The data and analyses in this report indicate that most of the classes that make up the targeted 
intervention had fairly high levels of fidelity to the READ 180 model. This conclusion is 
supported by Scholastic’s presentation of a report to MCS after Year 3, which indicated that, 
according to Scholastic’s observations and metrics, almost all classes were at least 90 percent on 
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model during Year 3. Because the implementation ratings conducted by RBS have increased 
from Year 3 to Year 4, it is reasonable to speculate that if Scholastic had conducted a Year 4 
evaluation of implementation fidelity, they would again conclude that most classes are 
substantially on model. Combined with the finding that in Year 4 there were again no significant 
differences between READ 180 students and the control group, there are at least three inferences 
that can be drawn from this: 


1. Improvements need to be made in recognizing and assigning those students who can 
benefit from READ 180 (i.e., reviewing research from other Striving Readers evaluations 
and other studies that indicate whether there are specific subgroups of students who 
might benefit from READ 180). 


2. Improvements are needed in measuring the quality and level of implementation (for 
example, increasing the number, depth, and breadth of observations, teacher self-reports, 
and other data sources and/or exploring the use of new statistical models for attributing 
fidelity to achievement). 


3. One or two years of participation in the READ /80 program did not help struggling 
readers more than other instruction or interventions used by MCS during the time of the 
study. 


Description of the Counterfactual and Development of the Intent-to-Treat 
Sample for the Targeted Intervention 


The targeted intervention is supplemental, so all students identified as the target population 
are also enrolled in a “regular” language arts class, whether or not they were selected to be 
enrolled in READ 180. Treatment students in three schools receive a truncated period for 
language arts: they are enrolled in a two-hour class that combines 90 minutes of READ 180 with 
30 minutes of language arts instruction, while control students in these schools are enrolled in 
“regular” language arts classes that last 45 to 55 minutes. 


Students in both the treatment and control groups might also participate in additional classes 
related to language arts. Some of these are reading classes, and this is made clear in the class 
name. Table III-7 provides a list of all of the classes related to reading and ELA in which 
evaluators found evidence of enrollment of either treatment or control students, sorted by grade. 
Class registration data were collected to detail how many treatment and control students were 
assigned to these classes. The shaded rows show the enrollment of students, by experimental 
condition, in READ /80 during Year 4: there was no evidence that any control students were 
enrolled in READ 180. 


Tables III-8 to III-19 describe the steps that evaluators took to arrive at the samples used for 
the Intent-to-Treat (ITT) analyses that are included in Section IV. As shown in the tables, all 
students in the enrollment files provided by MCS who had enrolled by a specific date early in the 
school year were included in the overall sample (for example, students who were enrolled by 
August 28, 2009, were included in the Year 4 analyses). Of these students, those who had valid 
pretest and posttest scores and about whom evaluators received demographic data were included 
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in the final analyses. Because of substantial changes made to the TCAP between Years 3 and 4, 
and because of the lack of statistically significant findings, not all tests were analyzed for all 
groups and years; therefore some analysis groups include all TCAP tests, some include only 
TCAP Reading/LA, and some include not TCAP tests. For detailed explanations of the analyses, 
please see section IV. 


The numbers in the counterfactual table differ from the numbers in the flow chart tables 
describing ITT samples because of the more specific nature of the counterfactual information. 
The ITT samples include all students that have pretest and posttest scores and the demographic 
variables. However, the counterfactual numbers only include those students whom RBS can link 
(via course registration files) to a specific READ /80, reading, or ELA teacher for at least half of 
the school days between pre- and posttest. Finally, the counterfactual numbers should not be 
added together because students might take more than one of these classes. 
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Table II1I-7: Numbers of ITT Students Enrolled for 50+ Days in Reading and ELA-Related Courses, by School and 


Experimental Condition 


Data sources: Year 4 enrollment data, Year 4 course enrollment data, and READ 180 random assignment data 


School A School B School C School D School E School F School G School H 

Grades _ READ _ READ _ READ _ READ _ READ _ READ _ READ 
Course Name Offered | Control 180 Control 180 | Control 180 | Control 180 | Control 180 | Control 180 | Control 180 
Creative Writing 6-90 Gage «| a 
Creative Writing 7-180 7 | 279 | 
Creative Writing 7-90 7 | | | | 5 | 0 
Language Arts 06 6 | 28 22/59 31/13 17 | 10, 19 | 56 36 | 10 23 33 | 44 20 
Language Arts 07 Ged. | Boo oe sap ge) aan tee ail pee | el od) sad d7 ||| B44) dz 
Language Arts07Honors | 7 | 2 0 | 4 OO | 2. (| os 2 ss | | 
Reading 6-180 6 lee Sar | eee |e a a 10 1 0 | 23 2414 
Reading 7-180 7 43-| 0° | 39 1 4 2 | Or 1436). 400)! 6 1 9 | 2 2 
Scholastic Read 6-180 6 Om 94 Boellieodt Mie) iets atetulerosd mos 23 33 0 
Scholastic Read 7-180 7 OQ = 2p 26 0 9 0 15 0 13 14 17 14 
Word Bldg Expl 6-90 6 caer 
Word Study Skills 6-180 6 | 6 | 13 
Content Area Read 6-180 | 6 55 13 
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Table III-8: Analysis of Sample Size for One-Year Impact of READ 180 on Spring 2007 Scores—Sixth- through Eighth-Grade 
Students, Year 1 


TCAP ITBS 


Total in ITT Group 


Total with (A), (B), and (C)' 664. 972 


511 712 


*For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was used as a covariate (e.g., TBS Total Reading). 
’The numbers shown in this row indicate the number of records that were used in the impact models. 
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Table III-9: Analysis of Sample Size for One-Year Impact of READ 180—Sixth-Grade Students, Year 1 


.. Reading/LA 
Trt Cnt 


TCAP 


Vocabulary 


Total in ITT Group 


Unknown 


Total with (A), (B), and (C)' 228 


372 | 


652 222 


* For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was used as a covariate (e.g., TBS Total Reading). 
’The numbers shown in this row indicate the number of records that were used in the impact models. 
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Table III-10: Analysis of Sample Size for One-Year Impact of READ 180—Sixth-Grade Students, Year 2 


TCAP 


Vocabulary 


Total in ITT Group 


Total with (A), (B), and (C)* 278 390 


278 390 


272 «387 ~=—«272 


380 


204 295 


*For each outcome measure (e.g., TBS Total Reading, the same type of pretest measure was used as a covariate, e.g., TBS Total Reading). 
"The numbers shown in this row indicate the number of records that were used in the impact models. 


204 299 
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Table IlI-11: Analysis of Sample Size for One-Year Impact of READ 180—Sixth-Grade Students, Year 3 
TCAP 


Ll Reading/LA 
Trt Cnt 


Soc. Studies 


Vocabulary 


Total in ITT Group 


Total with (A), (B), and (C)* 


*For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was used as a covariate (e.g., [TBS Total Reading). 
’The numbers shown in this row indicate the number of records that were used in the impact models. 
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Table I1I-12: Analysis of Sample Size for One-Year Impact of READ 180—Sixth-Grade Students, Year 4 


Total in ITT Group 


Total with (A), (B), and (C)'# 


= TOP | BS 
JReading® | Total Reading Comprehension _ vocabulary | 
Trt Cnt Trt Cnt Trt Cnt Trt Cnt 
238259 | 238 259 238289298259. 
211234 | fot 24 1922491 2nd 
i a 
0 0 31 24 31 24 31 24 
i es 
= 2 
3 es 
z a7 6 | OO 
| 298 269 | 215 242 216 242218243 
fs ee 
I OO | 23 a7 a8 
| 298 259 | 238 259 288 259288259. 
211 234 174 208 176 208 174 209 


* For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was used as a covariate (e.g., TBS Total Reading). 
’ The numbers shown in this row indicate the number of records that were used in the impact models. 
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Table IlI-13: Analysis of Sample Size for One-Year Impact of READ 180— 
Sixth-Grade Students, Years 1 through 4 


ITBS 


Total in ITT Group 


Total with (A), (B), and (C)' 


757 959 


1034 1264 


781 975 


766 967 


* For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was 
used as a covariate (e.g., TBS Total Reading). 

"The numbers shown in this row indicate the number of records that were used in the impact 
models. The current sample size is slightly smaller than the cumulative sample size of the past 
four years because students who were retained in grade have been removed from the current 


analyses. 
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Table IlI-14: Analysis of Sample Size for One-Year Impact of READ 180— 
Sixth- through Eighth-Grade Students in Year 1 and Sixth-Grade Students 
in Years 2-4 


Total in ITT Group 


112 160 


Unknown 


Total with (A), (B), and (C)' 1090 1384 1119 1405 


* For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was 
used as a covariate (e.g., TBS Total Reading). 

"The numbers shown in this row indicate the number of records that were used in the impact 
models. The current sample size is slightly smaller than the cumulative sample size of the past 
four years because students who were retained in grade have been removed from the current 
analyses. 
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Table IlI-15: Analysis of Sample Size for Two-Year Impact of READ 180—Seventh- and Eighth-Grade “Stayers,” Year 2 
TCAP 


Total in ITT Group 


Total with (A), (B), and (C)' 


* For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was used as a covariate (e.g., TBS Total Reading). 


’The numbers shown in this row indicate the number of records that were used in the impact models. 


Memphis Striving Readers/RBS: Evaluation and Impact Report 


June 17, 2011—Page 38 of 114 


Table III-16: Analysis of Sample Size for Two-Year Impact of READ 180—Seventh-Grade Students, Year 2 


TCAP 


Vocabulary 


Total in ITT Group 


Total with (A), (B), and (C)* 


193 


274 


193 


274. 189 271 


188 269 


145 


207 


* For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was used as a covariate (e.g., TBS Total Reading). 
"The numbers shown in this row indicate the number of records that were used in the impact models. 


145 208 


_ 155 
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Table IlI-17: Analysis of Sample Size for Two-Year Impact of READ 180—Seventh-Grade Students, Year 3 
TCAP 


Vocabulary 


Total in ITT Group 


Total with (A), (B), and (C)* 


* For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was used as a covariate (e.g., TBS Total Reading). 


’The numbers shown in this row indicate the number of records that were used in the impact models. 
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Table III-18: Analysis of Sample Size for Two-Year Impact of READ 180—Seventh- 


Grade Students, Year 4 


TCAP 


ITBS 


Total in ITT Group 


Total with (A), (B), and (C)' 171141 


140 123 


151 


126 


141 


* For each outcome measure (e.g., [TBS Total Reading), the same type of pretest measure was used as a 
covariate (e.g., [TBS Total Reading). 
"The numbers shown in this row indicate the number of records that were used in the impact models. 


Memphis Striving Readers/RBS: Evaluation and Impact Report 


June 17, 2011—Page 41 of 114 


Table III-19: Analysis of Sample Size for Two-Year Impact of READ 180— 
Seventh- and Eighth-Grade Students in Year 2 and Seventh-Grade Students 
in Years 3 and 4 


Total in ITT Group 


Unknown 


Total with (A), (B), and (C)" 


* For each outcome measure (e.g., TBS Total Reading), the same type of pretest measure was 
used as a covariate (e.g., [TBS Total Reading). 

"The numbers shown in this row indicate the number of records that were used in the impact 
models. The current sample size is slightly smaller than the cumulative sample size of the past 
four years because students who were retained in grade have been removed from the current 
analyses. 
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IV. Evaluation of the Impacts of the Targeted 
Intervention: Years 1 through 4 


This section includes descriptions of the study design, the student samples used, and the 
impacts of the targeted intervention—READ /80—on student outcomes. Each subsection 
includes information for all four years in chronological order. 


Study Design 


This subsection describes the design of the evaluation of the impacts in Years | through 4 of 
READ 180 on the quantitative outcome measures, the TCAP and ITBS, described in Section II. 
It includes descriptions of the study design, the student samples used, and how the outcome data 
were collected. (Year 4 TCAP data were not released until January 2011, and analyses were not 
completed in time for this report. Spaces are left in the appropriate tables to add the data in a 
revised version of the report.) 


Sample Selection 


Year I. At the beginning of Year 1, students who were struggling readers were identified in the 
sixth to eighth grades in each participating school. The intent was to randomly assign 
approximately 40 struggling readers in each grade in each school to receive READ 1/80 services. 
Students scoring in the lowest quartile on the state reading test were considered struggling 
readers and were eligible to be assigned to either the treatment or the control group. Before 
random assignment, the participating schools were given the opportunity to opt students out of 
being eligible (e.g., due to parent objections, assignment to self-contained classrooms for special 
education services, or teacher judgments that TCAP scores were not representative of students’ 
higher achievement levels).* 


Struggling readers who were not opted out and who had not received READ 180 services in 
the previous two school years were deemed eligible for random assignment to READ 180. 
Within each grade at each school, 40 students were randomly selected from this eligible pool to 
receive READ 180 services. These 40 students were enrolled by the school in one of two or 
three READ /80 classes in each grade. The students who were not randomly selected to receive 
READ 180 services were assigned to the control group. 


Due to significant variations in the way each school would or would not permit students 
receiving special education services to be assigned to READ 180, it was decided to base the 
READ 180 impact analyses on treatment and control students not receiving special education 
services. Students assigned to either condition that were enrolled in a Striving Readers school 
for more than 50 percent of instructional days between fall and spring ITBS administration 


* Most opt-outs occurred prior to random assignment. There were a few treatment group students who were opted 
out after random assignment. Since the control group counterparts for those students could not be identified, the 
treatment group students who were opted out after random assignment were retained in the treatment group for the 
ITT analyses. 
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constituted the intent-to-treat (ITT) group for the analyses of READ 180 impact on student 
outcomes in Year 1. Table IV-1 describes the number of treatment, control, and non-eligible 
students enrolled in the eight MSRP schools at the time of random assignment (September 18, 


2006). 
Table IV-1: Numbers of Students Enrolled in Striving Reader Schools in Year 1 by 
READ 180 Design Group 
READ 180 READ 180 
Treatment Control Non-Eligible Total MSRP School 
Grade Group* Group* Students Enrollment 
6 239 392 903 1,540 
7 233 370 1,270 1,880 
8 226 280 1,253 1,767 
All 698 1,042 3,426 5,187 


Data source: MCS enrollment files, 2006—2007 
*These two groups constitute the ITT group (N = 1,740) for READ 180 one-year impact analyses in Year 1. 
‘Enrollment as of 9/18/06 


Years 2 through 4. At the beginning of each year, struggling readers were identified in sixth 
grade in each participating school. As in Year 1, the intent was to randomly assign 
approximately 40 struggling readers in sixth grade in each school to receive READ 180 services. 
Before random assignment, the participating schools were again given the opportunity to opt out 
students from being eligible, and students who had received READ 180 services in the previous 
two years also were excluded. 


Within sixth grade at each school, between 30 and 40 students were randomly selected from 
this eligible pool to receive READ 180 services.” These students were enrolled by the school in 
one of one to three READ 180 classes in each grade. The students who were not randomly 
selected to receive READ 180 services were assigned to the control group. (As in Year 1, 
students receiving special education services were excluded from the impact analyses.) These 
sixth-grade students in each year constitute the intent to treat (ITT) groups for the analyses of the 
impact of one year of READ /80 on student outcomes in Years 2 through 4. Additionally, these 
sixth-grade students are included, along with all students who were included in the Year | ITT 
group, in the overall analysis of one-year impact on all students who received one year of 
treatment by the end of Year 4. Table IV-2 shows the number of treatment, control, and non- 
eligible sixth-grade students enrolled in the eight MSRP schools at the time of random 
assignment in Years 2, 3 and 4. 


Due to declining enrollments in several of the schools, it was not possible to assign 40 students to READ 180 and 
still have a reasonable number of control students (at least half the number of READ /80 students). 
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Table IV-2: Numbers of Sixth-Grade Students Enrolled in Striving Reader Schools in 
Years 2 through 4, by READ 180 Design Group 


READ 180 READ 180 
Treatment Control Non-Eligible Total MSRP School 
Year Group* Group* Students Enrollment 
2 289 404 734 1,427 
3 274 223 833 1,330 
4 238 259 761 1,258 


Data source: MCS enrollment files, 2007-2008, 2008-2009, and 2009-2010 

*These pairs of groups constitute the ITT groups for READ 180 one-year impact analyses in Year 2 (N = 693), 
Year 3 (N = 497), and Year 4 (N = 497). 

‘Enrollment as of 9/7/07 (Year 2), 8/28/08 (Year 3), and 8/28/09 (Year 4) 


In order to evaluate the impact of two years of READ 1/80 services, students from the Year 1 
ITT analysis group who were enrolled in the seventh and eighth grades in a participating school 
more than 50 percent of instructional days between fall and spring ITBS administration in Year 2 
were identified.'? Similarly, students who were in the sixth-grade ITT analysis groups during 
Years 2 and 3 and were enrolled in a participating school in seventh grade for more than 50 
percent of the instructional days between the fall and spring ITBS administrations during the 
following school year were identified. Table IV-3 describes the numbers of these students 
compared to the original number assigned to READ 180 treatment and control. The rates of 
attrition, while substantially lower between Years 2 and 3 than between Years | and 2 or Years 3 
and 4, are relatively consistent across experimental groups and grades. 


Table IV-3: Numbers and Attrition Rates of Students in Two-Year Impact Analysis 
Groups, by Grade and Design Group 


Grade 
During During During During First year Second year Attrition 
Year 1 Year 2 Year 3 Year4 Design Group of study* of study’ Rate 
a a. ane: READ A? 2 = 239 FOO) as 33% 
sus Oa tasers Taare arama i 260 34% 
a ——— EAD POO  —— es 32% 
7 8 Control 370 241 35% 
eae Oe a uN I OO OOO 
6 7 Control 404 364 10% 
area ee =e ea ee ee Se 
6 7 Control 223 141 37% 


Data sources: MCS enrollment files, 2006-2007, 2007—2008, 2008-2009, and 2009-2010 
* Enrollment as of 9/18/06, 9/25/07, and 9/25/08, respectively 
’ Enrollment as of 5/12/08, 5/6/09, and 5/6/10, respectively 


'° This identification process was consistent with an earlier procedural decision to include in the ITT analyses only 
those students who were enrolled in participating schools at the time of the spring administration of the ITBS. 
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Data Collection 


As described in Section II, the measures of student outcomes for determining the impact of 
READ 180 on struggling readers are the ITBS and the TCAP. In Years | through 3, standard 
scores were used to measure reading comprehension, vocabulary, and total reading on the ITBS. 
These scores are vertically equated across grade levels, so students in higher grades achieve 
higher scores on average. In Year 4, NCEs were used. NCEs provide an equal interval scale 
similar to standard scores, but they are not vertically equated. Instead, they represent 
achievement relative to students in the same grade. NCEs between 1 and 50 represent below- 
grade average achievement levels and NCEs between 50 and 99 represent above-grade average 
achievement levels. The reported internal consistency measures of reliability for these three 
scores for test levels administered to students in sixth through eight grades are high: 0.87—0.88 
for vocabulary, 0.90—0.92 for comprehension, and 0.94 for total reading.11 Scale scores were 
used to measure student achievement in reading/language arts, mathematics, science, and social 
studies on the TCAP. The reading/language arts and mathematics scale scores are vertically 
equated. However, the science and social studies scale scores are not (the distribution of scores 
in sixth through eighth grades have similar means and standard deviations). Efforts to obtain 
information on the psychometric properties of the TCAP were unsuccessful. 


The ITBS was administered twice in Year 1—during the week beginning September 18, 
2006, and during the week beginning April 30, 2007—by classroom teachers to all students in 
the MSRP schools, except those in self-contained special education classrooms and a very small 
number whose parents did not consent to the student’s participation in the testing. The spring 
2007 test scores measured treatment and control student reading achievement levels at the end of 
Year 1. The fall 2006 test scores were used to control for random differences in reading 
achievement levels between treatment and control students at the beginning of the year, as well 
as reduce the within-school error variance in the spring 2007 test scores. 


The ITBS was also administered twice in Year 2—during the weeks beginning September 
17, 2007, and May 12, 2008—by classroom teachers in MSRP schools. The fall administration 
was only for sixth-grade students; all students in MSRP schools took the spring administration. 
The spring 2008 test scores measured treatment and control student reading achievement levels 
at the end of Year 2. The fall 2007 test scores were used as a control variable for analyses of the 
impact of one year of READ 180 on the Year 2 achievement of sixth-grade students. The fall 
2006 test scores were used as control variables for analyses of the impact of two years of 
READ 180 on the Year 2 achievement of seventh- and eighth-grade students. 


The ITBS again was administered twice in Year 3—during the weeks beginning September 
8, 2008, and May 4, 2009—by classroom teachers in MSRP schools. As in Year 2, the fall 
administration was only for sixth-grade students; all students in MSRP schools took the spring 
administration. The spring 2009 test scores measured treatment and control student reading 
achievement levels at the end of Year 3. The fall 2008 test scores were used as a control variable 
for analyses of the impact of one year of READ 1/80 on Year 3 achievement of sixth-grade 


'' See the ITBS Guide to Research and Development (2003), pp. 71-73. 
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students. The fall 2007 test scores were used as control variables for analyses of the impact of 
two years of READ 1/80 on the Year 3 achievement of seventh-grade students. 


The ITBS again was administered twice in Year 4—during the weeks beginning September 
14, 2009, and May 3, 2010—by classroom teachers in MSRP schools. As in Year 3, the fall 
administration was only for sixth-grade students; all students in MSRP schools took the spring 
administration. The spring 2010 test scores measured treatment and control student reading 
achievement levels at the end of Year 4. The fall 2009 test scores were used as a control variable 
for analyses of the impact of one year of READ 180 on Year 4 achievement of sixth-grade 
students. The fall 2008 test scores were used as a control variable for analyses of the impact of 
two years of READ 1/80 on the Year 4 achievement of seventh-grade students. 


The TCAP is administered by MCS for the state in or about the first week in April each year. 
The spring 2007 test scores measured treatment and control student achievement levels in the 
four core content areas at the end of Year 1. The spring 2006 scores in the same content area 
were used to control for random treatment-control differences and reduce within-school error 
variance in spring 2007 scores. The spring 2007, spring 2008, and spring 2009 test scores, 
respectively, were used as control variables for analyses of the impact of one year of READ 180 
on the Year 2, Year 3, and Year 4 achievement of sixth-grade students. The spring 2006 test 
scores were used as a control variable for analyses of the impact of two years of READ 180 on 
the Year 2 achievement of seventh- and eighth-grade students. The spring 2007 test scores were 
used as a control variable for analyses of the impact of two years of READ 180 on the Year 3 
achievement for seventh-grade students. The spring 2008 test scores were used as a control 
variable for analyses of the impact of two years of READ /80 on the Year 4 achievement for 
seventh-grade students. 


Data Analysis 


ITT impact analyses of student achievement in reading and the four core content areas were 
conducted to assess the effects of one year of READ /80 on student outcomes for the 1,740 
eligible struggling readers in the sixth, seventh, and eighth grades in Year 1. Also, to investigate 
any suggestions of interactions of READ 180 impact and grade level, separate analyses were 
conducted for students in the sixth, seventh, and eighth grades. Separate analyses of the 693 
eligible sixth-grade students in Year 2, the 497 in Year 3, and the 497 in Year 4 were conducted 
to determine whether the impact of one year of READ /80 varied in Years | through 4. Finally, 
analyses of all ITT samples combined from all grades in Year 1 and sixth grade in Years 2 
through 4 were conducted to obtain an overall impact of one year of READ 180. 


Similar analyses were conducted to assess the effects of participating in READ 180 for two 
years on student achievement at the end of Year 2 for the remaining 820 ITT students in the 
seventh and eighth grades, at the end of Year 3 for the remaining 615 ITT students in seventh 
grade, and at the end of Year 4 for the remaining 312 ITT students in seventh grade. Separate 
analyses were conducted for the ITT students receiving two years of READ /80 in each year and 
a combined analysis for the 1747 students in all three years. 
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Multi-level regression analysis models were used to estimate and test the statistical 
significance of the difference between the achievement of students receiving READ 1/80 and the 
control group. Two-level models were employed for the Year 1 analyses that express the spring 
ITBS and TCAP scores as a function of student and school variables.'” The spring 2007 ITBS 
and TCAP scores were the dependent variables. The 2006 ITBS and TCAP scores— 
representing the same test or subject as the dependent variable—were included as the principal 
student-level covariate. Other control variables at the student and school levels were tested for 
inclusion as covariates in these analyses. The READ 180 treatment variable was included at the 
student level of these models. 


Similar two-level models were employed for the cross-sectional analyses of student 
achievement at the end of Years 2, 3, and 4. The only differences were the use of spring 2008 
(Year 2) or 2009 (Year 3) or 2010 (Year 4) ITBS and TCAP’® test scores as the dependent 
variables and, for the sixth-grade students, the use of fall 2007 (Year 2) or 2008 (Year 3) or 2009 
(Year 4) ITBS and spring 2007 (Year 2), spring 2008 (Year 3), or spring 2009 (Year 4) TCAP 
test scores as one of the student covariates. The complete specification of the multi-level 
regression models employed to determine the one- and two-year impacts of the READ 180 
intervention is provided in Appendix D. 


Table IV-4 summarizes the dependent and independent variables and the covariates included 
in these analyses. 


Selection of Covariates 


There are different approaches to including and/or excluding covariates in multi-level 
regression, as there are in single-level regression analyses. The approach that was used in these 
analyses was to (1) include all student- and school-level covariates in the model, (2) run the 
model, (3) eliminate the school covariate with the lowest significance level (highest p-value) not 
less than 0.2, (4) repeat steps two and three until the remaining covariates had p-values less than 
0.2, and (5) repeat steps two to four for the student covariates. 


Treatment of Missing Data 


The only variables in these analyses that contained missing data were the ITBS and TCAP 
scores. The number of students for whom test scores were missing in any analysis was small 
relative to the total number of students; therefore, simply omitting these students would not 
significantly compromise the power of these analyses. Thus, procedures for imputing missing 
values were not employed. The amount of attrition due to missing test scores and possible 
differential attrition between treatment and control groups were studied. The results of this study 
are presented in the discussion of the results of the READ 180 impact analyses below. 


'? Three-level models employing school, teacher, and student variables were explored. These analyses proved to be 
relatively complex and equivocal due to each student’s having different teachers for the core content areas and 
significant amounts of missing teacher data. Also, the results did not vary noticeably from the results of the two- 
level models. The evaluation team decided to omit three-level models from future impact analyses. 

' Spring 2010 TCAP results were made available in January 2011. These results will be included in the revised 
version of this report. 
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Table IV-4: All Variables Included in READ 180 Impact Analytical Models for Years 1 
through 4 


Variable Level Coding / Range 


Dependent 


Spring 2006/2007/2008/2009 TCAP Social Studies’ | Student 


African American Student 


Hispanic Student 


English Language Learner (Fall Student 


2006/2007/2008/2009)' 


Enrolled in Grade 7 in Year 1 Student 


Enrolled in Grade 8 in Year 1/Year 2 Student 


_Percentage African American (Fall 2006) == School O=100 

Percentage Special Ed (Fall2006) = School, «S 0-100, 

Percentage FRL'(Fall2006¢) = School 100 

Percentage ELL’ (Fall2006) School 100 
School Enrollment (Fall 2006) School 400-1200 


Data sources: ITBS and TCAP files, school enrollment and demographic data files, and READ 180 random 
assignment files, 2006 through 2010 

*Second through fourth dates apply for analyses in Years 2 through 4. 

"Second through fourth dates apply for analyses of sixth-grade students in Years 2 through 4. 

? Students receiving free or reduced-priced meals 

SEnglish Language Learners 
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Description of the Samples of Students for READ 180 Impact Analyses in 
Years I through 4 


Equivalence on Student Demographic Characteristics 


Year I. As described above, 1,740 eligible struggling readers were randomly assigned to the 
READ 180 treatment or control groups in Year | of the MSRP study. The grade level and other 
demographic characteristics of these students are presented in Table IV-S. 


Table IV-5: Significant Differences in Demographic Characteristics of Analytic Groups 


Student Characteristic Control READ 180 Sig. Level Total 

Year 1, All Grades Sample 

Enrolled in Grade 6 392 37.6% 239 34.2% 631 36.3% 
Enrolled in Grade 7 370 35.5% 233 33.4% 0.05 603 34.7% 
Enrolled in Grade 8 280 26.9% 226 32.4% 506 29.1% 
African American 955 91.6% 657 94.1% 0.05 1612 92.6% 
Hispanic 86 8.2% 40 5.7% 0.05 126 7.2% 
English Language Learner 83 8.0% 34 4.9% 0.01 117 6.7% 
Year 2, Sixth-Grade Sample 

Female 169 41.8% 143 49.5% 0.05 312 45.0% 
Male 235 58.2% 146 50.5% 381 55.0% 
Year 3, Sixth Grade Sample 

African American 204 91.5% 265 96.7% 0.01 469 94.4% 
Hispanic 18 8.1% 9 3.3% 0.02 27 5.4% 
English Language Learner 16 7.2% 7 2.6% 0.02 23 4.6% 
Year 4, Sixth Grade Sample 

African-American 241 93.1% 232 97.5% 0.02 473 95.2% 
Hispanic 18 6.9% 6 2.5% 0.02 24 4.8% 
English Language Learner 20 1.7% 6 2.5% 0.01 26 5.2% 
Years 1-4, Sixth Grade (only) Sample 

African-American 1174 92.9% 992 95.9% 0.00 2166 94.3% 
Hispanic 88 7.0% 41 4.0% 0.00 129 5.6% 
English Language Learner 81 6.4% 35 3.4% 0.00 116 5.0% 
Years 1-4; All Students with One Year of Treatment 

African-American 1768 92.4% 1422 95.2% 0.00 3190 93.6% 
Hispanic 143 7.5% 70 4.7% 0.00 213 6.3% 
English Language Learner 136 7.1% 61 4.1% 0.00 197 5.8% 
Years 1-4; All Students with Two Years of Treatment (“Stayers”) Sample 

English Language Learner 66 7.2% 33 4.8% 0.05 99 6.2% 


Data sources: MCS enrollment files, 2006-2007, 2007-2008, 2008-2009, and 2009-2010; Grade 6 SR Students in 
Yr 4 R180 Impact Analyses.sav; Grade 6 SR Students in Yr 1 2 3 and 4 R180 Impact Analyses.sav; All Students in 
Grade 6 for Yrs 1+2+3+4 and Grade 7 and 8 for Yr 1 Analyses of One Year Impact.sav; Grade 7 Students in Yr 4 
Impact Analyses - Stayers.sav; and All Students for Yrs 2+3+4 R180 Analyses of Two Year Impact 

Notes: Numbers in parentheses are the numbers of students in each group who have valid test scores. The current 
sample size is slightly smaller than the cumulative sample size of the past four years because students who were 
retained in grade have been removed from the current analyses. 
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The number of students decreases as the enrollment grade increases, and this difference is 
reflected more strongly in the control group since approximately equal numbers were randomly 
assigned to the treatment group in each grade. These differences in grade enrollment between 
treatment and control groups emphasize the importance of treating the student’s enrollment grade 
as a covariate in the analyses of READ /80 impact for students from more than one grade. Also, 
all but two students were either African American or Hispanic, which supported the creation of 
two dichotomous covariates to represent membership in these two race/ethnicity groups. Finally, 
the differences in demographic composition of the treatment and control groups were relatively 
minor, although some were statistically significant given the large number of students overall. 
Including these characteristics as student-level covariates in the analytical models helps to 
control for these small differences, as well as reduce the within-school error variance in the 
dependent variables. Tables detailing comparisons of all of the demographic characteristics in 
all years are included as Appendix E. 


Equivalence on Baseline Achievement 


With random assignment, the baseline scores of students in treatment and control groups 
should be very similar on all seven tests. To determine whether this was true, researchers 
compared the baseline achievement of treatment and control groups for all students in Year 1 
and, in subsequent years, for incoming sixth graders. Specifically, comparisons between 
treatment and control groups on the baseline 2006 ITBS and TCAP test scores were carried out 
for the 1,740 students in the Year 1 ITT sample and the 820 Year 2 “stayers.” Treatment and 
control comparisons were made on the baseline 2007 ITBS and TCAP test scores for the Year 2 
sixth-grade ITT sample, on the baseline 2008 ITBS and TCAP test scores for the Year 3 sixth- 
grade ITT sample, and on the baseline 2009 ITBS and TCAP test scores for the Year 4 sixth- 
grade ITT sample. Analyses confirmed that while there were slight differences between groups 
on the different tests in different years, there were statistically significant differences between the 
groups only in Year 2. Table IV-6 shows these differences and their significance levels. Tables 
detailing all of the scores and differences, for all seven tests in all four years, together with their 
significance levels, are included as Appendix F. 


Table IV-6: Significant Differences in Baseline Achievement Scores of Analytic Sample 
Groups 


Year 2, Sixth-Grade Students Means 

Test Score Control READ 180 Sig. Level 
ITBS Total Reading Standard Score 184.9 (364) | 182.0 (247) | 0.007 
ITBS Comprehension Standard Score 182.8 (365) | 179.7 (247) | 0.014 
TCAP Reading/LA Scale Score 487.8 (403) | 482.4 (289) | 0.004 


Data sources: ITBS and TCAP, 2007 


Note: Numbers in parentheses are the numbers of students in each group who have valid test scores. 
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Impact of Participation in READ 180 for One Year on Student 
Achievement in Years 1 through 4 


One-Year Impact of READ 180 on Students in Sixth through Eighth Grades Combined in 
Year 1 


The multi-level regression model described in Appendix D was used to estimate the one-year 
impact of READ 180 on the reading and content area achievement of the 1,740 eligible 
struggling readers in the Year 1 READ ISO ITT sample. Table IV-7 summarizes the results of 
these analyses for student reading achievement measured by the ITBS and student achievement 
in the four core content areas measured by the TCAP. (The complete results of the multi-level 
analyses of the READ 180 impact on these seven test scores can be found in Appendix G in 
Tables G-1—G-7.) 


Table IV-7 displays several statistical parameters. The unadjusted means show the actual 
mean 2007 test scores for the treatment and control groups. The numbers in parentheses at the 
bottom of these cells is the number of students in the respective group with a valid test score. 
The adjusted means are the average scores controlling for all covariates retained in the 
analytical model—the variable indicating treatment/control group membership and all 
“significant” covariates (p < 0.2). The estimated impact is the difference between the treatment 
and control group adjusted means (treatment minus control). A positive impact means the 
READ 180 treatment group averaged higher achievement on the particular test than the control 
group, controlling for covariates included in the final analytical model (see Appendix G, Tables 
G-1-G-7). A negative impact means the control group averaged higher than the treatment group. 
The significance level and effect size are two indicators of the importance of the estimated 
difference. Conventionally, a significance level less than 0.05 is an acceptable indication that the 
estimated difference is not due to chance, i.e., that it is “statistically significant.” 

Conventionally, an effect size between 0.2 and 0.5 is considered small, between 0.5 and 0.8 is 
medium, and greater than 0.8 is large. 


The estimated READ 180 impacts for all seven test scores are quite small—less than one 
standard/scale score unit. None is statistically significant (p < 0.05); and all effect sizes are quite 
small. There is no reason, therefore, to reject the hypothesis that the average achievement of the 
treatment and control groups was the same at the end of Year 1. In other words, participation in 
READ 180 did not have a significant impact on student achievement levels in reading or in the 
four core content areas at the end of the first year of the study. 
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Table IV-7: One-Year Impact of READ 180 in Year 1 on Spring 2007 Scores on Each 
Achievement Test—Sixth- through Eighth-Grade Students 


Unadjusted Means Adjusted Means Est. Effect Signif. 
Test Score Control | READ 180 | Control | READ 180 Impact Size* Level 
Heat iege oy a 192.6 192.1 -0.5 -0.03 0.532 
Hee eh aise a es 187.0 187.0 0.0 0.00 0.976 
Ce a ne 197.5 197.6 0.1 0.01 0.937 
eee a es ea 496.9 497.1 0.2 0.01 0.882 
ae ee oe a 500.0 500.2 0.2 0.01 0.904 
neo las fae as 185.6 185.1 -0.5 -0.03 0.573 
sea Eas ee ee eas) 185.0 185.8 0.8 0.05 0.323 


Data sources: ITBS and TCAP, 2006-2007 
*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 


adjusted mean test scores divided by the control group’s test score standard deviation. 


"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 2006 
administrations and the spring 2007 administrations. 


One-Year Impact of READ 180 on Students in Each Grade in Year 1 


The analyses of READ 180 impact reported above were based on the total ITT sample of 
students in sixth through eighth grades. The same analyses were also carried out separately for 
students in each of these three grades. In the Year 4 report, Year | sixth-grade results can be 
compared with results for sixth graders in the Years 2-4 to see if there are any changes over time 
in the impact of READ 180. The results for seventh- and eighth-grade students in Year | allow a 
comparison of the impact of this targeted intervention across grades. The calculation of the one- 
year impact of READ 180 for seventh- and eighth-grade students in Years 2 through 4 is not 
relevant due to their previous participation in READ 180. Table IV-8 presents the results of the 
analyses of the one-year impact of READ 180 on sixth-grade students. 
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Table IV-8: One-Year Impact of READ 180 in Year 1 on Spring 2007 Scores on Each 
Achievement Test—Sixth-Grade Students 


Unadjusted Means Adjusted Means Est. Effect Signif. 

Test Score Control READ 180 Control READ 180 Impact Size* Level 
Hite aie ke Bent ata) 183.8 184.3 0.5 0.04 0.665 
ad cea estas aan eas 180.3 186.9 -1.0 -0.07 0.441 
Aaenaapecenies en ibe) 186.8 189.5 2.7 0.17 0.056 
— eae ap es 488.9 488.9 0.0 0.00 0.996 
aed cin ee mee 492.3 494.7 2.4 0.08 0.279 
nee Sts ae ne 186.6 187.6 1.0 0.07 0.423 
ae say ue 6) as 186.7 187.5 0.8 0.04 0.615 


Data sources: ITBS and TCAP, 2007 

*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 

"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 2006 
administrations and the spring 2007 administrations. 


The estimated READ 180 impacts in the sixth grade in Year | were not as small as they were 
for all grades. The impacts on the ITBS vocabulary subtest and TCAP mathematics were greater 
than two standard/scale score units. Again, however, none of the impacts are statistically 
significant (p < 0.05), although the impact on ITBS vocabulary approaches this level and the 
associated effect size, 0.17, is close to the small range of 0.2—-0.5. Overall for sixth grade, there 
is no reason to reject the hypothesis that the average achievement of the treatment and control 
groups was the same at the end of Year 1. Participation in READ 180 did not have a significant 
impact on student achievement levels in reading or in the four core content areas in the sixth 
grade in Year 1. 


Similar analyses carried out for seventh- and eighth-grade students yielded non-significant 
estimated impacts with effect sizes under 0.2. The complete results of the multi-level analyses of 
the Year 1 READ 180 impact on these seven test scores for each grade can be found in Appendix 
G in Tables G-8-G-28. 


One-Year Impact of READ 180 on Sixth-Grade Students in Year 2 


The one-year impact of READ 180 on ITBS and TCAP test scores for sixth-grade students in 
Year 2 was determined using the same multi-level model (see Appendix D) for the 693 sixth- 
grade students in the Year 2 ITT sample. The only differences were that the dependent variables 
were the 2008 ITBS and TCAP test scores and the respective test score covariate was from the 
2007 administrations. The results of these analyses are summarized in Table IV-9. The 
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complete results of the multi-level analyses of the Year 2 READ 180 impact on these seven test 
scores for sixth grade can be found in Appendix G in Tables G-29-G-35. 


The estimated READ 180 impacts in the sixth grade in Year 2 were of a similar size as they 
were in Year 1. There were more negative impacts favoring the control group. However, even 
the largest estimated impact, for TCAP mathematics, was not statistically significant (p < 0.05). 
Overall for sixth grade, there is no reason to reject the hypothesis that the average achievement 
of the treatment and control groups was the same at the end of Year 2. Participation in 
READ 180 did not have a significant impact on student achievement levels in reading or in the 
four core content areas in the sixth grade in Year 2. 


Table IV-9: One-Year Impact of READ 180 in Year 2 on Spring 2008 Scores on Each 
Achievement Test—Sixth-Grade Students 


Unadjusted Means Adjusted Means Est. Effect Signif. 

Test Score Control READ 180 | Control READ 180 | Impact Size* Level 
Linen coe pest os 184.4 183.7 -0.7 -0.06 0.468 
ae ose) ie 181.8 180.0 -1.8 -0.12 0.170 
sete peste oa oe 187.4 186.8 -0.6 -0.04 0.639 
ae Area as oa 494.7 496.5 1.9 0.06 0.407 
eee es 00) nen 500.0 495.8 -4.2 -0.15 0.070 
eed ug ee - 186.5 186.3 -0.2 -0.01 0.876 
ecm a news faa) te 185.8 183.5 -2.3 -0.15 0.087 


Data sources: ITBS and TCAP, 2007—2008 

*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 

"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 2007 
administrations and the spring 2008 administrations. 


One-Year Impact of READ 180 on Sixth-Grade Students in Year 3 


The one-year impact of READ 180 on ITBS and TCAP test scores of sixth-grade students in 
Year 3 was determined using the same multi-level model (see Appendix D) for the 497 sixth- 
grade students in the Year 3 ITT sample. The only differences were that the dependent variables 
were the 2009 ITBS and TCAP test scores and the respective test score covariate was from the 
2008 administrations. The results of these analyses are summarized in Table IV-10. The 
complete results of the multi-level analyses of the Year 3 READ 1/80 impact on these seven test 
scores for sixth-grade students can be found in Appendix G in Tables G-36—G-42. 


The estimated READ 180 impacts in the sixth grade in Year 3 were, with one exception, of a 
similar size as in Years | and 2. The impact on TCAP Reading/LA scale scores was larger, 
favoring the READ 180 group, and statistically significant (p < 0.05). The effect size was in the 
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small range, however, and there appears to be an unusually large adjustment downward of the 
control group mean (from 479.5 to 473.8). Although no explanation for this uncharacteristically 
large adjustment could be found, one should use caution in interpreting this significant impact. 
The other six impacts favored the control group and were not statistically significant. With the 
one tentative exception, there is again no reason to reject the hypothesis that the average 
achievement of the sixth grade treatment and control groups was the same at the end of Year 3. 


Table IV-10: One-Year Impact of READ 180 in Year 3 on Spring 2009 Scores on Each 
Achievement Test—Sixth-Grade Students 


Unadjusted Means Adjusted Means Est. Effect | Signif. 

Test Score Control _| READ 180 | Control | READ 180 | Impact | Size* | Level 
Hails pica ae oon 184.2 182.2 -2.0 | -0.18 | 0.081 
aaa cer oie oer A a ook 179.6 178.2 -1.4 | -0.09 | 0.314 
ee shay i oA (308) 188.9 186.6 2.3. | -0.16 | 0.111 
has oo on . oes, 473.8 480.7 6.9 | 0.21 | 0.030 
ae Or a . es) 491.7 488.5 3.2 | -0.11 | 0.215 
heat Sl ar ea 183.3 182.4 -0.9 | -0.06 | 0.550 
ee Ria one a i oes) 183.6 183.0 0.6 | -0.04 | 0.714 


Data sources: ITBS and TCAP, 2008-2009 

*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 

"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 2008 
administrations and the spring 2009 administrations. 


One-Year Impact of READ 180 on Sixth-Grade Students in Year 4 


The one-year impact of READ 180 on ITBS and TCAP" test scores of sixth-grade students 
in Year 4 was determined using the same multi-level model (see Appendix D) for the 497 sixth- 
grade students in the Year 4 ITT sample. The only differences were that the dependent variables 
were the 2010 ITBS and TCAP test scores and the respective test score covariate was from the 
2009 administrations. Also, analyses were conducted in NCEs in Year 4 to facilitate cross-site 
analyses by the Striving Reader Evaluation Technical Assistance contractor. The results of these 
analyses are summarized in Table [V-11. The complete results of the multi-level analyses of the 
Year 4 READ 180 impact on the three ITBS test scores for sixth-grade students can be found in 
Appendix G in Tables G-43—G-45. 


'4 Because of substantive changes in the TCAP between Years 3 and 4, and because there were no significant 
impacts on TCAP Reading/LA or ITBS scores in Year 3, researchers analyzed only the TCAP Reading/LA scores 
in Year 4. 
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The estimated READ 180 impacts in the sixth grade in Year 4 on ITBS NCE scores were not 
statistically significant (p < 0.05). Effect sizes were all smaller than 0.20 (in absolute value). 
There is again no reason to reject the hypothesis that the average achievement of the sixth grade 
treatment and control groups was the same at the end of Year 4. 


Table IV-11: One-Year Impact of READ 180 in Year 4 on Spring 2010 Scores on Each 
Achievement Test—Sixth-Grade Students 


Unadjusted Means Adjusted Means Est. Effect | Signif. 

Test Score Control | READ 180 | Control | READ 180 | Impact | Size* Level 
Hess potab Reading ee aan 18.8 19.3 0.5 | 0.05 | 0.621 
ee: Comprehension oe ae 24.1 23.1 2.0 | 0.17 | 0.089 
(ee ao aa 211 | 205 | -06 | -0.05 | 0.583 
eae en ony 703.4 | 7015 | -19 | -0.07 | 0.433 


Data sources: ITBS and TCAP, 2009-2010 

*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 

"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 2009 
administrations and the spring 2010 administrations. 


One-Year Impact of READ 180 on Sixth-Grade Students, Years 1-4 Combined 


The one-year impact of READ 180 on ITBS and TCAP'” test scores of sixth-grade students 
in Years | through 4 was determined using the same multi-level model (see Appendix D) for the 
2298 sixth-grade students in the combined four ITT samples. The dependent variables were the 
2007-2010 ITBS and TCAP test scores and the respective test score covariate was from the 
2006-2009 administrations. Also, analyses were conducted in NCEs to facilitate cross-site 
analyses by the Striving Reader Evaluation Technical Assistance contractor. The results of these 
analyses are summarized in Table [V-12. The complete results of the multi-level analyses of the 
READ 180 impact on the three ITBS test scores for sixth-grade students in Years 1-4 can be 
found in Appendix G in Tables G-46-G-48. 


The estimated READ 180 impacts in the sixth grade across Years | through 4 on ITBS NCE 
scores were not statistically significant (p < 0.05). Effect sizes were all very small. There is no 
reason to reject the hypothesis that the average achievement of the sixth grade treatment and 
control groups was the same for all four ITT samples combined. 


'S Because analyses of one year of treatment showed no significant differences between treatment and control 
groups on the TCAP Reading/LA test in Years 1, 2, 3, or 4, researchers determined that a combined analysis of 
TCAP Reading/LA scores was not warranted. 
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Table [V-12: One-Year Impact of READ 180 in Years 1—4 on Spring 2007-2010 Scores on 


Each Achievement Test—Sixth-Grade Students 


Unadjusted Means Adjusted Means Est. Effect | Signif. 

Test Score Control | READ 180 | Control | READ 180 | Impact | Size* Level 
ioe neta eae Meee ee 19.4 19.1 -0.3 | -0.03 | 0.530 
tee eel ae a 22.1 22.0 0.1 | -0.01 | 0.814 
aco yoeabulaty en aes 21.0 21.0 0.0 | 0.00 | 0.975 


Data sources: ITBS and TCAP, 2007-2010 

*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 

"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 
2006-2009 administrations and the spring 2007-2010 administrations. 

Note: The current sample size is slightly smaller than the cumulative sample size of the past four years because 
students who were retained in grade have been removed from the current analyses. 


One-Year Impact of READ 180 on Sixth-Grade Students in Years 1-4 plus Seventh and 
Eighth Grades in Year 1 


The one-year impact of READ 180 on ITBS"® test scores of all students in all four years of 
the study who had completed one year of READ 180 (sixth-grade students in Years | through 4 
and seventh- and eighth-grade students in Year 1) was determined using the same multi-level 
model (see Appendix D) for the 3,407 students in the combined six ITT samples. The dependent 
variables were the 2007-2010 ITBS and TCAP test scores, and the respective test score covariate 
was from the 2006-2009 administrations. Also, analyses were conducted in NCEs to facilitate 
cross-site analyses by the Striving Reader Evaluation Technical Assistance contractor. The 
results of these analyses are summarized in Table I[V-13. The complete results of the multi-level 
analyses of the one-year READ 180 impact on the three ITBS test scores for all students 
randomly assigned to treatment or control can be found in Appendix G in Tables G-49-G-51. 


The estimated READ 180 one-year impacts on all students in the combined six ITT samples 
from Years 1-4 on ITBS NCE scores were not statistically significant (p < 0.05). Effect sizes 
were very small. There is no reason to reject the hypothesis that the average achievement of the 


treatment and control groups after the treatment group received one year of READ /80 was the 
same. 


'° Because analyses of one year of treatment showed no significant differences between treatment and control 
groups on the TCAP Reading/LA test in Years 1, 2, 3, or 4, researchers determined that a combined analysis of 
TCAP Reading/LA scores was not warranted. 
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Table [V-13: One-Year Impact of READ 180 in Years 1—4 on Spring 2007-2010 Scores on 
Each Achievement Test—All Students in Year 1 and Sixth-Grade Students in Years 2—4 


Unadjusted Means Adjusted Means Est. Effect | Signif. 

Test Score Control | READ 180 | Control | READ 180 | Impact | Size* Level 
ioe are re eae Hiceo) 20.1 19.7 -0.4 -0.04 0.322 
Hess Per ies ae 22.6 22.5 -0.1 -0.01 0.790 
aco pocket es) ee 21.8 21.6 -0.2 | -0.02 | 0.590 


Data sources: ITBS and TCAP, 2007-2010 

*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 

"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 
2006-2009 administrations and the spring 2007-2010 administrations. 

Note: The current sample size is slightly smaller than the cumulative sample size of the past four years because 
students who were retained in grade have been removed from the current analyses. 


Impact of Participation in READ 180 for Two Years on Student 
Achievement in Years 2 through 4 


Two-Year Impact of READ 180 on Seventh- and Eighth-Grade Students in Year 2 


In order to determine the impact of participating in READ /80 for two years at the end of 
Year 2, a two-level model similar to the model described in Appendix D was used to estimate the 
difference between treatment and control students on spring 2008 ITBS and TCAP test scores, 
controlling for their 2006 baseline scores, along with other student-level covariates. Since the 
students in these analyses were the 820 “stayers” in the seventh and eighth grades in Year 2, only 
one dummy variable indicating which students were in eighth grade was employed to control for 
within-school differences attributable to the student’s grade level. The school-level covariates 
remained the same. Table IV-14 presents the results of these analyses of the two-year impact of 
READ 180. The complete results of the multi-level analyses of the two-year READ 180 impact 
in Year 2 on these seven test scores for seventh- and eighth-grade students can be found in 
Appendix G in Tables G-52-G-58. 
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Table IV-14: Two-Year Impact of READ 180 in Year 2 on Spring 2008 Scores on Each 
Achievement Test—Seventh- and Eighth-Grade Students Combined 


Unadjusted Means Adjusted Means Est. Effect Signif. 

Test Score Control READ 180 Control READ 180 Impact Size* Level 
ene soe nee 198.0 198.1 0.1 0.01 0.967 
Ee ceo ota he ae 193.7 193.0 -0.7 -0.03 0.639 
Waenaapecenies Gon ee 201.5 203.3 1.8 0.08 0.280 
ek eae nen ne 505.3 506.7 1.4 0.05 0.446 
aed ee se) ae 511.9 511.5 -0.4 -0.01 0.871 
ete St ies ae 184.4 184.7 0.3 0.02 0.782 
Le say ee a ae) 186.5 185.1 -1.4 -0.11 0.129 


Data sources: ITBS and TCAP, 2006 and 2008 
*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 


"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 2006 


administrations and the spring 2008 administrations. 


The estimated impacts of two years of participation in READ 180 on the “stayers” in the 
seventh and eighth grades in Year 2 were not statistically significant (p < 0.05). There is no 


reason to reject the hypothesis that the average achievement of the treatment and control groups 


was the same at the end of Year 2. Participation in READ 180 for two years did not have a 
significant impact on student achievement levels in reading or in the four core content areas at 


the end of Year 2. 


Similar analyses carried out separately for each of the two grades yielded similar non- 


significant estimated impacts with one exception. In seventh grade, there was one significant 
difference in favor of the treatment group for ITBS vocabulary test scores. However, at a 
significance level of 0.05, this result may have also been due to chance since one would expect 
five percent of all hypothesis tests to reject the null hypothesis of no difference when the null 
hypothesis is true. The complete results of the multi-level analyses of the READ 180 impact on 
these seven test scores for the “stayers” in the seventh and eighth grades, by grade, can be found 
in Appendix G in Tables G-59-G-72. 


Two-Year Impact of READ 180 on Seventh-Grade Students in Year 3 


The impact of participating in READ /80 for two years was examined again in Year 3, 
looking only at students in the seventh grade. The eighth-grade students had received two years 
of READ /80 in sixth and seventh grade, and since the targeted intervention was intended as a 
two-year intervention, their involvement in the research study was ended. Consequently, the 
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results based on eighth-grade students MCS may have enrolled in READ 180 for a third year 
were omitted. 


The same two-level model was used to estimate the difference between treatment and control 
students on spring 2009 ITBS and TCAP test scores, controlling for the 2007 baseline scores of 
the seventh-grade students, along with the other student-level covariates. Table IV-15 presents 
the results of these analyses of the two-year impact of READ /80 on seventh-grade students. 

The complete results of the multi-level analyses of the READ 180 two-year impact on these 
seven test scores for the “stayers” in the seventh grade can be found in Appendix G in Tables 
G-73-G-79. 


Table IV-15: Two-Year Impact of READ 180 in Year 3 on Spring 2009 Scores on Each 
Achievement Test—Seventh-Grade Students 


Unadjusted Means Adjusted Means Est. Effect | Signif. 

Test Score Control | READ 180; Control | READ 180 | Impact Size* Level 
eae crane i oon oe 194.9 194.7 -0.2 -0.01 0.910 
Misch a sata Bony Hae 191.0 188.6 -2.4 -0.11 0.266 
emi cpanaty nee pee 198.6 199.9 1.3 0.07 0.487 
eae Riad iD) nee, 497.0 495.8 -1.2 -0.04 0.629 
sea oN aos oa 508.2 505.5 -2.7 -0.08 0.281 
oe oo iets nN 185.2 183.0 -2.2 -0.13 0.149 
el — eee au a 186.8 185.5 -1.3 -0.11 0.202 


Data sources: ITBS and TCAP, 2007 and 2009 

*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 

"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 2007 
administrations and the spring 2009 administrations. 


The estimated impacts of two years of participation in READ 180 on the “stayers” in the 
seventh grade in Year 3 were not statistically significant (p < 0.05). There is no reason to reject 
the hypothesis that the average achievement of the treatment and control groups was the same at 
the end of Year 3. Participation in READ /80 for two years did not have a significant impact on 
student achievement levels in reading or in the four core content areas at the end of Year 3. 


Two-Year Impact of READ 180 on Seventh-Grade Students in Year 4 


The impact of participating in READ /80 for two years was examined again in Year 4, 
looking only at students in the seventh grade. The same two-level model was used to estimate 
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the difference between treatment and control students on spring 2010 ITBS and TCAP"’ test 
scores, controlling for the 2008 baseline scores of the seventh-grade students, along with the 
other student-level covariates. Analyses were conducted in NCEs to facilitate cross-site analyses 
by the Striving Reader Evaluation Technical Assistance contractor. Table IV-16 presents the 
results of these analyses of the two-year impact of READ /80 on seventh-grade students. The 
complete results of the multi-level analyses of the READ 1/80 two-year impact on the three ITBS 
test scores for the “stayers” in the seventh grade can be found in Appendix G in Tables G-80— 
G-82. 


Table IV-16: Two-Year Impact of READ 180 in Year 4 on Spring 2010 Scores on Each 
Achievement Test—Seventh-Grade Students 


Unadjusted Means Adjusted Means Est. Effect | Signif. 

Test Score Control | READ 180 | Control | READ 180 | Impact | Size* | Level 
Nee eee ea ao) 20.3 21.5 1.2 0.10 0.364 
Hes Seo a ae en 23.0 25.4 2.4 0.22 0.064 
ice eee a aan 22.1 22.1 0.0 0.00 0.999 
eae tad a an 699.8 699.2 -0.6 -0.02 0.837 


Data sources: ITBS and TCAP, 2008 and 2010 

*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 

"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 2008 
administrations and the spring 2010 administrations. 


The estimated impacts of two years of participation in READ 180 on the “stayers” in the 
seventh grade in Year 4 were not statistically significant (p < 0.05). However, the impact on 
comprehension scores approached significance and the effect size was 0.22. There is no reason 
to reject the hypothesis that the average achievement of the treatment and control groups was the 
same at the end of Year 4. Participation in READ 180 for two years did not have a significant 
impact on student achievement levels in reading at the end of Year 4. 


Two-Year Impact of READ 180 on Seventh-Graders in Years 2—4 Plus Eighth-Graders in 
Year 2 


The impact of participating in READ /80 for two years was examined for the “stayers” in all 
ITT samples, i.e., seventh- and eighth-grade students at the end of Year 2 and seventh-grade 
students at the end of Years 3 and 4. The same two-level model was used to estimate the 


'” Because of substantive changes in the TCAP between Years 3 and 4, and because there were no significant two- 
year impacts on TCAP Reading/LA or ITBS scores in Year 3, researchers analyzed only the TCAP Reading/LA 
scores in Year 4. 
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difference between treatment and control students on spring 2008-2010 ITBS'* test scores, 
controlling for the 2006—2008 baseline scores, along with the other student-level covariates. 
Analyses were conducted in NCEs to facilitate cross-site analyses by the Striving Reader 


Evaluation Technical Assistance contractor. Table IV-17 presents the results of these analyses of 


the two-year impact of READ /80 on all “stayers.” The complete results of the multi-level 
analyses of the READ 180 two-year impact on these seven test scores for all “stayers” can be 
found in Appendix G in Tables G-83-—G-85. 


Table IV-17: Two-Year Impact of READ 180 in Years 2 through 4 on Spring 2008-2010 
Scores on Each Achievement Test—Seventh- and Eighth-Grade Students in Year 2 and 


Seventh-Grade Students in Years 3 and 4 


Unadjusted Means Adjusted Means Est. Effect | Signif. 

Test Score Control | READ 180 | Control | READ 180 | Impact | Size* Level 

7 oa Total Reading ai ats 21.7 21.9 02 | 0.02 | 0.769 
4 a Romibpiehension ae ne 24.5 24.4 0.1 | -0.01 | 0.878 
4 = Nogao Waly ne a 22.5 23.2 0.7 | 0.06 | 0.294 


Data sources: ITBS and TCAP, 2006-2008 and 2008-2010 

*The method used to calculate effect size was Glass’ A, the difference between treatment and control groups’ 
adjusted mean test scores divided by the control group’s test score standard deviation. 

"Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline 
2006-2008 administrations and the spring 2008-2010 administrations. 


The estimated impacts of two years of participation in READ 180 on the “stayers” in Years 2 
through 4 were not statistically significant (p < 0.05). There is no reason to reject the hypothesis 
that the average achievement of the treatment and control groups was the same at the end of 
Years 2 through 4. Participation in READ 180 for two years did not have a significant impact on 
student achievement levels for all “stayers.” 


Differential Attrition in READ 180 One-and Two-Year Impact Analyses 


The numbers of students in the treatment and control groups in the analyses of one- and two- 
year impact analyses are smaller than the numbers in the corresponding comparisons of baseline 
achievement levels. This is because not all of the students with valid baseline scores also had 
valid scores from the spring 2007, 2008, and 2009 administrations at the end of Years 1, 2, and 
3, respectively. Although the levels of attrition for both the treatment and control groups might 
or might not be similar, the important issue is whether the same types of students, especially in 
terms of their achievement levels, were lost from both groups. If not, one could argue that the 
estimated impacts were biased. That is, the treatment group may have lost students that would 
have scored higher (or lower) than the students lost from the control group. 


'S Because analyses of one year of treatment showed no significant differences between treatment and control 
groups on the TCAP Reading/LA test in Years 1, 2, 3, or 4, researchers determined that a combined analysis of 
TCAP Reading/LA scores was not warranted. 
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This potential differential attrition was studied for the Year 1 ITT sample by comparing the 
average baseline 2006 test scores of the students who also had a spring 2007 score to the average 
baseline 2006 test scores of students without a spring 2007 score. This comparison was done for 
both the treatment and control groups.’ If this attrition effect were higher or lower in one group, 
this differential attrition would have to be acknowledged as possibly biasing the estimated 
impact of participation in READ 180. 


To illustrate how differential attrition was studied each year, the results of the study of 
differential attrition for the estimated Year 1 immediate impacts may be found in Appendix I in 
Table I-1. In summary, Table I-1 shows that the effects of attrition in both treatment and control 
groups on baseline 2006 test scores did not exceed one standard or scale score point, and no 
differential treatment effects were statistically significant (p < 0.05). It seems reasonable to 
conclude that differential attrition was not a biasing factor affecting the interpretation of the 
estimated Year 1 impacts of READ 180. 


The potential for differential attrition was also studied for the ITT samples of sixth-grade 
students in Years 2 through 4 by comparing the average baseline test scores of the students with 
and without spring test scores at the end of the year. For all three years, the interaction of 
experimental group and having spring test scores was insignificant, indicating no differential 
attrition between treatment and control groups. 


Finally, differential attrition was studied for “stayers” used in the analyses of two-year 
impacts. No evidence for differential attrition was found in each of the Years 2 through 4. 


A Special Analysis of Differential Impacts of READ 180 in MCLA and 
Non-MCLA Schools in Years I and 2 


The impact of MCLA on student outcomes is described later in this report. However, 
separate analyses were carried out with the READ /80 ITT samples to determine whether the 
one- and two-year impacts of READ 180 in Years | and 2 were different in the MCLA treatment 
schools than in the MCLA control schools. (Analyses of the combined effects of MCLA and 
READ 180 ended after Year 2 because the MCLA experimental research condition ended when 
the whole-school intervention moved from Cohort 1 to Cohort 2 schools.) 


The dependent and independent variables and the covariates for addressing the research 
questions about the interaction of READ 180 and MCLA are the same as those described in the 
above analyses of READ 180 impacts, with one exception. An independent variable representing 
the participation of schools in the MCLA treatment was included in the analytical models for 
these analyses. This variable was included at the school level since schools were randomly 
assigned to the MCLA treatment or control condition. This model is specified in Appendix D. 


'° The analytical method was a univariate ANOVA of baseline 2006 test scores, employing a 2x2 factorial design 
crossing the READ 180 treatment/control condition with possession (yes/no) of a spring 2007 test score. The 
interaction of these two factors was tested for significance to determine whether or not there was a differential 
attrition effect. 


Memphis Striving Readers/RBS: Evaluation and Impact Report June 17, 2011—Page 64 of 114 


The results of the analyses of the seven spring 2007 test scores for the READ 180 ITT sample 
in Year | are presented in Table IV-18. The unadjusted and adjusted means are presented for the 
four combinations of READ 180 treatment/control and MCLA treatment/control conditions. The 
estimated interaction effect is the difference between the estimated READ 180 impact in MCLA 
treatment and control schools. A positive interaction effect means that the READ 180 impact 
was larger in MCLA control schools; a negative one means the READ /80 impact was larger in 
MCLA treatment schools. 
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Table IV-18: Interaction of READ 180 and MCLA Year 1 Impacts on Spring 2007 Scores on Each Achievement Test 


Unadjusted Means Adjusted Means Est 
READ 180 Control READ 180 Control lnteraction 
MCLA | Control MCLA_ | Control Effect* Effect | Signif. 
Test Score MCLA | Control | MCLA Control (A) (B) (C) (D) (A-C)-(B-D) Size Level 

ITBS Total Reading 191.2 193.7 | 191.0 192.7 
Standard Sears (231) (280) (371) (341) 193.6 190.6 193.4 191.3 0.9 0.06 | 0.518 
ITBS Comprehension 187.8 187.5 185.9 187.5 

189.7 184. 188.4 185.7 2.7 14 16 
Standard Score (236) (281) (374) (344) : a : pee 
ITBS Vocabulary 196.1 200.2 | 196.3 197.8 

197.4 197.7 197.6 197.2 -0.7 0 75 
Standard Score (234) (285) (381) (345) : wee ua 
TCAP Reading/LA 499.1 497.0 | 496.0 495.6 

496.6 | 497.7 494.8 499. A % .24 
Scale Score (311) | (353) | (512) (460) ee s Re eva 
TCAP Mathematics 496.1 506.8 | 498.4 501.7 

495.9 04.4 498.8 01.9 -5.4 0.15 .058 
Scale Score (310) | (351) | (511) (460) 2 ze : : 
TCAP Science 184.9 | 186.2 | 185.3 184.9 

184.2 186.0 185.5 185.9 -1.4 0.09 | 0.388 
Scale Score (305) | (338) | (503) (412) 
TCAP Social Studies 185.1 186.9 | 184.6 185.7 

185.2 186.5 184.4 185.7 01 95 
Scale Score (306) | (338) | (495) (411) oe nN oe Hone 


Data sources: ITBS and TCAP, 2006 and 2007 
*The formula for the estimated interaction effect reads, “The difference between the READ 180 impact in MCLA schools and the READ 180 impact in non-MCLA 


schools.” 
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Figure IV-1 illustrates the magnitude and direction of these interactions. For example, on the 
ITBS Total Reading measure, the READ 180 impact for the MCLA schools is 0.9 points higher 
(0.2— (-0.7)) in the MCLA schools than it is in the non-MCLA or control schools. 


Figure [V-1: READ 180 Year 1 Impact in MCLA Treatment and Control Schools 
3.0 


SMCLA 
®Control 


READ 180 Impact 


T T T T T T 1 


ITBS Total ITBS Comp ITBS Vocab TCAPR/LA TCAP Math TCAP Sci TCAP Socst 


Outcome Measure 
Data sources: ITBS and TCAP, 2006 and 2007 


These interaction effects range between -5.4 and 3.1, an impact that was slightly more than 
five scale-score points larger on spring 2007 TCAP mathematics scores in the MCLA control 
schools and three-scale score points larger on TCAP reading/language arts scores in MCLA 
treatment schools. Clearly these results are mixed, although the READ /80 impact is larger in 
MCLA treatment schools on reading measures (except vocabulary) and larger in MCLA control 
schools in the other content areas. However, none of the interaction effects is statistically 
significant (p < 0.05), although the effect for the TCAP mathematics scores approached this 
level. Also, all of the effect sizes are less than 0.2. In summary, it seems reasonable to conclude 
that the READ 180 impact on student achievement did not vary significantly between MCLA 
treatment and control schools in Year 1. 


The interaction of the READ 180 and MCLA impacts was also analyzed for each grade 
separately. The results for the sixth grade are presented in Table IV-19. Again, the results were 
mixed and non-significant for the most part. The READ /80 impact on the spring 2007 TCAP 
science scores was significantly larger in MCLA control schools than in MCLA treatment 
schools (p < 0.05), and the effect size was 0.32. The larger READ 1/80 impact on ITBS 
vocabulary scores in MCLA treatment schools was not statistically significant, but the effect size 
was ().21. It is interesting to note that the pattern of (non-significant) larger READ 180 impacts 
in the MCLA treatment schools for reading measures and larger impacts in the control schools 
for non-reading measures was also present in the sixth-grade results. 
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Table IV-19: Interaction of Year 1 READ 180 and MCLA Impacts on Spring 2007 Scores on Each Test—Sixth Grade 


Unadjusted Means Adjusted Means Est 
READ 180 Control READ 180 Control Interaction 
MCLA Control MCLA Control Effect* Effect | Signif. 
Test Score MCLA Control MCLA Control (A) (B) (C) (D) (A-C)-(B-D) Size Level 
ITBS Total 185.7 184.3 183.4 183.3 
Reading (79) (99) (145) (142) 185.0 183.7 183.8 183.8 1.3 0.11 0.529 
ITBS 181.4 178.9 181.2 178.2 
F 180.4 178.4 181.2 179.4 0.2 0.01 0.947 
Comprehension (80) (99) (146) (142) 
ITBS Vocabulary 190.3 189.7 185.6 188.6 
191.2 188.0 186.7 186.7 3.2 0.21 0.258 
(80) (100) (146) (144) 
TCAP 491.3 488.0 489.4 486.4 
Reading (110) (118) (189) (183) 488.4 489.5 487.8 490.2 1.3 0.05 0.781 
TCAP 493.6 496.1 497.2 491.4 
: 493.6 495.4 494.3 490.1 -6.0 0.19 0.166 
Mathematics (110) (117) (189) (183) 
TCAP 186.2 189.5 187.0 184.8 
Shinde (108) (113) (187) (165) 188.3 186.9 189.9 183.1 -5.4 0.32 0.037 
TCAP 186.9 187.6 187.4 184.5 
Social Studies (109) (113) (184) (165) 186.7 187.6 187.6 185.0 -3.5 0.19 0.240 


Data sources: ITBS and TCAP, 2006 and 2007 


*The formula for the estimated interaction effect reads, “The difference between the READ 180 impact in MCLA schools and the READ 180 impact in non-MCLA 


schools.” 


Separate analyses conducted for the seventh- and eighth- 
grade students also yielded only a few interactions worth noting, 
but they did all favor the READ 180 impact in MCLA treatment 
schools. Two statistically significant (p < 0.05) interactions were 
found that supported a larger READ /80 impact in MCLA 
treatment schools on ITBS comprehension scores for eighth- 
grade students (effect size = 0.33) and on TCAP reading/LA 
scores for seventh-grade students (effect size = 0.40). Another 
two interactions had effect sizes slightly above 0.20, but were not 
statistically significant (p < 0.05). These two suggested a larger 


READ 180 impact in MCLA treatment schools on TCAP 
reading/LA scores for eighth-grade students and on TCAP 
science scores for seventh-grade students. The complete results 
of the multi-level analyses of the Year 1 READ 180 impact for 
MCLA treatment and control schools for students in all three 
grades and each grade separately can be found in Appendix I in 
Tables I-1—I-28. 
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Additional analyses of the interaction between READ 1/80 and MCLA were carried out on 
student outcome measures obtained at the end of Year 2. One set of seven analyses examined 
the interaction of the one-year impact of READ 180 and MCLA for the Year 2 sixth-grade ITT 
sample, looking at the spring 2008 ITBS and TCAP scores and controlling for the fall 2007 ITBS 
and spring 2007 TCAP scores. A second set examined the interaction of the two-year impact of 
READ 180 and MCLA for the Year 2 seventh- and eighth-grade “stayers,” looking at the 2008 
spring ITBS and TCAP scores controlling for the fall 2006 ITBS and spring 2006 TCAP scores. 
None of the interaction effects in these 14 analyses were statistically significant (p < 0.05). The 
complete results of these additional multi-level analyses can be found in Appendix I in Tables 
[-29-]-42. 


There do not appear to be any clearly interpretable patterns in the differences in READ 180 
impact for MCLA treatment and control schools. In Year | there was a suggestion of larger 
READ 180 impacts on reading measures in MCLA treatment schools versus larger impacts on 
non-reading measures in control schools in the sixth and seventh grade analyses as well as the 
analyses based on all grades. However, of the 42 interaction effects tested for Years | and 2, 
only three (7%) were statistically significant. Using a significance criterion of p < 0.05, five 
percent (or 2) of the 42 tests would be expected to be found significant by chance. 


Analyses of the interaction of READ 180 and MCLA impacts were not carried out in Years 3 
and 4 since the MCLA control schools are receiving the MCLA treatment in Years 3 and 4, thus 
making it impossible to estimate an MCLA impact. 


Conclusions Regarding All READ 180 Impact Analyses 


There was a lack of significant one-year impacts of participation in READ 180 in Years 1 and 
2. There was one small but significant one-year impact on sixth-grade students in Year 3—on 
the TCAP Reading/Language Arts measure. The lack of significant two-year impacts of 
participation in READ 180 was consistent in Years 2 and 3: none of the Year 2 or Year 3 two- 
year impacts was significant. There were no significant one- or two-year impacts on ITBS 
scores in Year 4, nor did any of the analyses of combined ITT samples yield significant one- or 
two-year impacts. Finally, the examination of the interaction between the impacts of READ 180 
and the whole-school intervention in Years | and 2 yielded no clearly interpretable patterns in 
the differences in the impact of READ 180 for MCLA treatment and control schools. 


Further Analyses 


In an earlier footnote, it was noted that three-level regression models were explored to 
include the characteristics of teachers linked to students for each core content area. These 
analyses were sufficiently complex and the results sufficiently equivocal to lead to the decision 
to omit their results from any further reporting. In addition, they did not result in different 
estimates of READ 1/80 impact. Therefore, it was also decided to omit analyses designed to 
determine if READ /80 impacts are moderated by teacher characteristics. (Moderation by school 
characteristics was not investigated in the two-level models due to the very low levels of 
between-school variation.) Secondary analyses were conducted to confirm further the results of 
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the Year 3 and 4 analyses, and findings from these analyses are included in the following 
subsection. 


Propensity Score Analyses of One-Year and Two-Year Impacts of 
READ 180 


To confirm the results of the impact analyses presented earlier in this section, the evaluators 
created matched samples of students using Propensity Score Matching (PSM) and then 
conducted independent-samples t-tests to compare outcomes across the matched groups. The 
analytic samples were composed of sixth-grade students who had received one year of treatment 
during any of the study years and of students who had received two years of treatment during 
any two of the study years. Because of wide variations in implementation fidelity during Year 1, 
data from this year were not included in these analyses. This provided a method to determine 
whether the implementation variations had enough impact to affect the entire samples. 


The outcome measures for these analyses are the same as those in the impact analyses: ITBS 
Total Reading, Comprehension, and Vocabulary. For the one-year analysis, the outcome 
variables are the scores achieved by students in the spring of their first (or only) year in 
READ 180. For the two-year analysis, the outcome variables are the scores achieved in the 
spring of the students’ second year in READ 180. 


The analytic sample for one year of treatment (NV = 1,667) consists of 795 treated students 
and 872 control students. Table IV-20 shows the distribution of the sample by year. There are 
507 cases with missing values in at least one variable. Multiple imputations (MI) were employed 
to replace the missing values on each variable with predicted values based on the observed data 
(Rubin, 1987; Schafer, 1997; Allison, 2001). In this study, the imputed value was the average of 
predicted values from five imputations, as recommended by Allison (2001). The MI approach 
not only helped retain the complete sample but also generated much less biased estimates of 
parameters and more accurate standard errors compared to deterministic imputation methods, 
such as mean substitution (Haitovsky, 1968; Allison, 2000). 


Table IV-20: Distribution of Students (All in Sixth Grade) in Treatment and Control 
Groups by Year—One-Year Impact Analysis 


Year Treatment Control 
2 283 390 
3 274 223 
4 238 259 
Total 795 872 


Data sources: MCS demographic and achievement test data files, Years 2 through 4 


The propensity score e(x) is the probability that a student with observed covariates X = x is 
assigned to the treatment group rather than the control group, which is computed using the 
following formula (Cox, 1970): 
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exp(a + B’ x) 
1+exp(a + Bp’ x) 


e(x) = P(S=1| X=x) 


In the equation above, S denotes the treatment (S = | if assigned to READ 180, and S = 0 if 
assigned to control), X denotes a set of observed covariates, e(x) or P(S = 1|X = x) denotes the 
probability of a student being assigned to READ 180 given a vector of observed covariates x, and 
exp (a+ B'x) denotes the exponential function of the coefficients in a fitted logistic regression. 
Note that the value of (@ + B'x) represents the estimated odds of a student being in READ 180. 
Following a strategy suggested by Rosenbaum and Rubin (1984), evaluators built a propensity 
score model using stepwise logistic regression with all covariates and their two-way interactions 
as candidate variables. The variables that were retained in the logistic regression model are the 
number of days a students was registered for ELA, the quadratic term of ELA days, the quadratic 
term of TCAP reading pretest scores, whether the student is African American, and ITBS Total 
Reading pretest scores. The optimal matching was applied using the “optmatch” package in R 
(Hansen & Klopfer, 2006) based on the estimated propensity score so that the average absolute 
propensity score difference between the matched pairs is minimized (Gu & Rosenbaum, 1993). 


Evaluators used the standardized bias (SB) introduced by Rosenbaum and Rubin (1985) to 
examine bias reduction before and after matching. A lower SB indicates that the matched 
samples are more similar than the original, randomly determined ITT samples. 


_ 100(% x 


treated 


2 2 
S treated +S control 


Table IV-21 shows the means, standard deviations, and standardized biases of both 
unmatched and matched samples across treatment and control groups for the one-year impact 
analysis. As shown by the table, the treated and control groups have noticeable differences on 
several observed covariates, such as proportion of English Language Learners (ELL), proportion 
of African Americans, number of days registered for ELA classes, ITBS Total Reading pretest 
scores (ITBS tot Pre), TCAP math pretest scores (TCAP Math Pre), and TCAP reading pretest 
scores (TCAP Read Pre). The absolute SBs of these covariates are larger than 10 before the 
matching. After the matching, the number of covariates with absolute SBs greater than 10 
dropped from six to one (i.e., days registered in ELA classes). The average absolute SB dropped 
from 6.63 to 2.61. Therefore, the treated and control groups are roughly similar after the 
matching. 


control ) 


SB 
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Table IV-21: Comparison of the Covariate Balance Between the Matched Pairs with the 
Balance between the Unmatched Treated and Control Groups in One-Year Treatment 
Data File 


Unmatched data Matched data 


Treated Control Treated Control 

(N = 795) (N = 872) (N =777) (N =777) 
Variable M SD mM so|ssB.{ M | sD M SD | SB. 
Gender 0.49 0.50| 0.46 0.50! 594] 049 0.50] 0.46 0.50] 4.38 
FRL 0.97 0.18] 0.96 0.20] 568| 0.97 0.17| 095 0.21] 7.41 
ELL 0.03 0.18] 0.06 0.24/-12.64] 0.03 0.18] 0.04  0.20| -2.71 
AfrAmer 0.96 0.19] 093 0.25] 1342] 0.96 0.19] 096 0.19] 0.00 
ELA Days 138.04 38.45| 144.33 26.52|-19.03 | 141.16 32.90] 144.30 27.12|-10.41 


ITBS Voc Pre 185.36 15.88 | 186.87 15.97 -9.45 | 185.47 | 15.92 | 185.64 15.96 -1.06 


ITBS Com Pre_ | 179.53 | 15.18 | 180.89 14.64 -9.09 | 179.66 | 15.26] 180.33 14.67 -4.42 


ITBS TotPre _| 182.47 12.41| 183.88 12.16 |-11.50 | 182.59 12.46] 182.99 12.07] -3.26 


TCAPMath Pre | 484.96 25.42| 487.55 24.12|-10.46 | 484.91 25.60| 486.72 24.32] -7.23 


TCAPRead Pre | 479.45 25.97| 482.81 22.96 |-13.71 | 479.51 26.06] 481.59 23.50] -8.39 


TCAPScie Pre | 180.36  16.85| 181.21 17.03| -5.01 | 180.40 16.88] 180.52 17.46| -0.73 


TCAPSoc Pre | 187.29 12.97] 188.33 13.00] -8.01 | 187.25 13.02] 187.66 13.16) -3.08 


Mean SB_ -6.63 Mean SB|_ -2.61 


Data sources: MCS demographic and achievement test data files, Years 2 through 4 
Note: SB = Standardized Bias 


Table IV-22 compares the differences between treated and control units in ITBS vocabulary, 
comprehension, and total reading posttest scores before and after the optimal matching. As 
shown by the table, before the matching the control group outperformed the treated group by 
1.16 points (p < .05) on the ITBS Total Reading posttest score. After the matching, the two 
groups had no significant differences in any of the three outcome variables. 


Table IV-22: Comparison of the ITBS Posttest Scores Between the Matched Pairs with the 
ITBS Posttest Scores between the Unmatched Treated and Control Groups in the One- 
Year Treatment Data File 


Unmatched Matched 

TES | | | Mean — | | _ Mean 
Posttest Group | N M___ SD __ Diff. p N  M__ SD __ Dfff. p 

T | 795 187.19 14.32 -1.21 09 | 777 187.31 14.66 | 
Vocabulary | -.56 45 

C 872 188.40 14.68. 777 187.87 14.37 
Compre- T | 795 | 179.65 | 14.73 -1.41) 13 | 777 | 179.71 | 14.78 | | 
hension Cc 872 180.76 14.97. 777 180.02 1470, | 8 
Total T | 795 183.42 11.95 -1.16 .049| 777 183.51 12.00 | 
Reading Cc | 872 184.58 11.95. | 777 | 18395 11820 


Data sources: MCS demographic and achievement test data files, Years 2 through 4 
Note: The p values are based on independent-samples t-tests for equality of means between the treated and control 
groups. 
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The analytic sample for two years of treatment (N = 1,720) consists of 574 seventh-graders 
and 159 eighth-graders in the treated group and 746 seventh-graders and 241 eighth-graders in 
the control group. Table IV-23 shows the distribution of the sample by year. There are 553 
cases that had missing values in at least one variable. As in the one-year impact analysis, 
multiple imputations were conducted to replace all missing values on each variable with 
predicted values from regression models based on the observed data. Evaluators again built a 
logistic regression model using stepwise selection of covariates (Rosenbaum & Rubin, 1984). 
The variables that were retained in the model include English Language Learners, the quadratic 
term of TCAP reading pretest scores, and whether the student received free or reduced-price 
lunch. 


Table IV-23: Distribution of Students in Treatment and Control Groups by Year— 
Two-Year Impact Analysis 


Year Seventh Grade Eighth Grade 
Treatment Control Treatment Control 
2 160 260 159 241 
3 243 345 0 0 
4 171 141 0 0 
Total 574 746 159 241 


Data sources: MCS demographic and achievement test data files, Years 2 through 4 


Table IV-24 shows that the standardized biases of all covariates in the two-year impact group 
are much smaller for the matched data than for the unmatched data. In particular, the average 
absolute SB decreased from 1.51 to 0.42. This indicates that the treated and control groups are 
much more similar after the optimal matching. Table [V-25 compares the differences between 
treated and control units in ITBS Vocabulary, Comprehension, and Total Reading posttest scores 
before and after the optimal matching. As shown by the table, the two groups had no significant 
differences in any of the three outcome variables before or after the optimal matching. This lack 
of differences confirms the findings of the ITT analyses presented earlier in this section. It also 
suggests that the wide variations in implementation fidelity during Year 1 did not wipe out 
effects that would have appeared in multiyear ITT analyses if Year 1 had been omitted. 
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Table IV-24: Comparison of the Covariate Balance Between the Matched Pairs with the 
Balance between the Unmatched Treated and Control Groups in Two-Year Treatment 


Data File. 
Unmatched data Matched data 
Treated Control Treated Control 
(N = 733) (N = 987) (N = 733) (N = 733) 
Variable M SD M SD | SB M SD M SD | SB 
Gender 0.47 0.50; 0.46 0.50] 2.14 | 0.47  0.50| 0.47. 0.50| -0.27 
FRL 0.93  0.26| 0.91 0.29] 7.77 | 0.93 0.26] 0.93  0.26| 0.00 
ELL 0.05 0.21} 0.07. 0.25} -9.91 0.05 0.21| 0.05. 0.21} 0.00 
AfrAmer 0.95 0.22) 0.93 0.26] 812 | 0.95 0.22} 0.95 0.22] 0.00 
ELA Days 0.22 041/ 0.24 0.43|-647 | 0.22 0.41/ 0.26 0.44| -9.62 
Grade 8 144.41 26.38| 144.38 24.35] 0.14 | 144.41 26.38] 144.53 24.43] -0.46 
ITBS Voc Pre__| 188.44 17.47| 188.88 17.45 | -2.57 | 188.44 17.47| 188.34 17.44| 0.52 
ITBS Com Pre_| 181.73 _16.02| 182.45 15.48 | -4.57 | 181.73 16.02| 181.98 15.02| -1.56 
ITBS Tot Pre | 185.11. 13.66| 185.67. 13.20] -4.21 | 185.11 13.66] 185.17 13.04] -0.46 
TCAPMath Pre | 486.64 24.74| 486.41 27.28| 0.88 | 486.64 24.74| 484.10 27.95| 9.62 
TCAPRead Pre | 477.59 26.98] 479.72 25.61| -8.07 | 477.59 26.98| 478.11 25.61 | -1.98 
TCAPScie Pre | 181.95 15.82] 182.10 16.30] -0.94 | 181.95 15.82| 181.39 16.26] 3.48 
TCAPSoc Pre | 187.44 15.73] 187.23 15.88] 1.36 | 187.44 15.73] 186.23 16.75| 7.48 
Mean SB_ -1.51 Mean SB__ 0.42 


Data sources: MCS demographic and achievement test data files, Years 2 through 4 
Note: SB = Standardized Bias 


Table IV-25: Comparison of the ITBS Posttest Scores Between the Matched Pairs with the 
ITBS Posttest Scores between the Unmatched Treated and Control Groups in Two-Year 
Treatment Data File. 


ITBS Group Unmatched Matched 
Posttest 
Mean Mean 
N  =M SD __ Diff. op | oN M __— SD___ Diff. p 
18 733. 196.01 16.12. 733 194.84 16.12. 
Vocabulary © 47 55 © 02 .98 
C 987 196.48 16.12. 733 194.82 16.13. 

: T 733 189.72 19.00 | 733 189.72 19.00 | 
els so 29 7 PEE 8340 
hension C 987 191.18 19.35. 733 190.55 18.93. 

T 733. 199.41 19.92. 733 199.41 19.92. 
forse. Sg) APE SOE gh ep el a 
Reading C 987 199.29 19.57 | 733 198.59 19.62 | 


Data sources: MCS demographic and achievement test data files, Years 2 through 4 
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V. Evaluation of the Implementation of the Whole- 
School Intervention, Year 4 


Summary of the Design 


Section V presents results from the evaluation of the Year 4 MCLA whole-school 
intervention, which was provided to teachers in Cohort 2 schools for the final two school years 
(Years 3 and 4) of the Memphis Striving Readers Project. During Years 1 and 2, developers 
provided the intervention to teachers at Cohort 1 schools, while Cohort 2 teachers served as 
research controls in experimental analyses. At the end of Year 2, the professional development 
intervention moved to Cohort 2 schools to offer teachers an opportunity to receive literacy 
coaching support and other program services. RBS evaluated aspects of program 
implementation, including participation rates, level of coaching support offered, attendance at 
evening course sessions, and the level of implementation fidelity to the original MCLA model 
designed by developers for the final year of operations. 


The following subsections detail the research questions addressed in the Year 4 
implementation evaluation of the whole-school intervention, contextual factors affecting teachers 
and students in all eight Striving Readers schools, and the information used to calculate ratings 
of implementation fidelity at the four Cohort 2 schools. 


The implementation evaluation of the MCLA whole-school intervention in Year 4 addressed 
two overarching research questions: 
1. What contextual district- and school-level factors influenced the implementation of the 
MCLA program? 
2. To what degree did the implemented MCLA treatment match the intended program 
standards and features? 


Specific research questions about the implementation of the whole-school intervention in 
Cohort 2 schools are as follows: 
¢ What was the Year 4 MCLA classroom instructional model? 


¢ What types and amounts of professional development were provided to teachers, 
principals, and literacy coaches? 


¢ What proportion of teachers received and participated at different levels of professional 
development (e.g., how many used program materials or completed the evening MCLA 
course)? 


¢ What types of coaching support was provided to teachers? 
¢ What was teachers’ level of program implementation? 
To address these questions, RBS collected information from program participants and MCLA 
experts in the following ways: First, evaluators conducted interviews with the eight (100%) 


principals at the Striving Readers schools and collected survey data from 214 (98.1%) teachers to 
provide insight into factors influencing MCLA implementation and for information about related 
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professional development events offered in the Cohort 2 schools. Second, evaluators reviewed 
information contained in course syllabi, MCLA instructor templates, transcripts of interviews 
with program experts and developers, and curricular resource center (CRC) inventory lists to 
address the research question about the extent to which the MCLA treatment as implemented 
matched the intended program model. Third, RBS examined MCLA course attendance sheets, 
coaching logs, and CRC checkout logs for answers about the types of professional development 
provided and levels of program participation. Fourth, RBS measured classroom implementation 
in Year 4 among Cohort 2 teachers through analysis of data collected using teacher surveys, 
weekly checklists, daily coaching logs, and direct observation. Evaluators also analyzed the 
coaching logs and teacher survey responses for further details about the type of coaching services 
that were provided to participants. 


Finally, as mentioned in Section I of this report, RBS also engaged the MCLA literacy 
coaches (who provided frequent, in-class support to teachers) in rating teachers’ level of 
implementation fidelity with respect to specific classroom practices. The Innovation 
Configuration (IC) Map (Hall & Hord, 2006) developed by the grant team provides operational 
definitions of program components at various levels of implementation. The teacher-related 
practices rated by coaches during a one-and-a-half-day retreat designed and facilitated by RBS 
included the following: 


* introducing literacy strategies to students 

* modeling how to use the strategies 

* providing instruction that is explicit and direct 

* differentiating instruction 

* enabling students to use strategies independently 


* revisiting strategies consistently during lessons 


Throughout the school year, coaches collected evidence about teacher implementation such 
as observation notes, student work products, and class handouts. Before using the IC Map to rate 
fidelity implementing the specific components, RBS asked the coaches to assign global ratings 
(i.e., low, medium, high) for an individual teacher based on their overall impression of the 
teacher’s highest level of implementing a composite comprising of all five components. Coaches 
drew upon their professional experience with teachers and data sources in each teacher’s 
portfolio and issued a separate rating for each component. A cluster analysis conducted by RBS 
evaluators showed significant agreement between coaches’ initial global ratings assigned to 
teachers and computer-generated clusters based on specific ratings of each component. Clusters 
indicated that the IC Map was useful in describing observable aspects of teaching and learning 
and sufficiently sensitive to distinguish between various levels of implementation fidelity. 


As part of the cross-site Striving Readers evaluation, RBS analyzed the information 
described in this section to address three overarching research questions: 


1. What was the level of implementation and variability of professional development for 
teachers, coaches, and principals? 
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2. What was the level of implementation and variability of classroom instruction? 
3. What did the counterfactual look like? 


Figure V-1 shows “at a glance” the type of information that was collected from each 
stakeholder group as part of the Year 4 cross-site Striving Readers evaluation. 


Figure V-1: Data Sources Linked to Striving Readers Research Questions—MCLA, Year 4 


Research Questions Measures/Data Sources 
Class 
Surveys/Logs Interviews Obs. | Record Review 
a ae 
e = gs 3 
oj = 5S 
- = a) eae Se) 8 
2 S o <= s 2 @ 22 -« a 
8 © 5 © 2 gS s | 8) 3 © 
® fo} oe fo} = x< g QD: oO oc 
What was the level of implementation and variability of professional development for teachers, 


coaches, and principals? 


Type/amount of PD provided to teachers | X | Xx | | | x | | x 
Proportion of teachers at different levels Xx | Xx Xx x 
of PD 
Proportion of teachers at adequate level Xx | Xx Xx xX 
of PD 
Types/amount of coaching provided to 
teachers 
Proportion of teachers at different levels Xx Xx 
of coaching 
Proportion of teachers at adequate level Xx Xx 
of coaching 
Type/amount of PD provided to coaches | x | Xx | 
Proportion of coaches at different levels 
xX =X 
of PD 
Type/amount of PD provided to school Xx Xx 
principals 
Proportion of school principals at different x Xx 
levels of PD 


What was the level of implementation and variability of classroom instruction? 
Proportion of teachers with access to 


; xX xX Xx 
materials and resources 
Proportion of teachers who implemented Xx Xx 
literacy strategies (CAPs) 
Proportion of teachers who implemented Xx Xx X Xx x x 
the model at adequate level 
What did the counterfactual look like? 
Proportion of teachers reporting literacy- x x 


related PD at follow-up 
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Contextual Factors in MSRP Schools 


RBS interviewed each of the eight building principals in spring 2010 about school district 
initiatives that might affect MCLA implementation or its sustainability. (See Appendix J for the 
principal interview guide.) Six principals identified district and state initiatives that had an 
impact on either supporting or serving as barriers to their efforts to improve students’ 
implementation of MCLA literacy strategies. Principals cited the following initiatives as either 
supporting or hindering MCLA implementation in their schools: 


* The presence of additional READ 180 teachers provided by the grant facilitated overall 
MCLA implementation; however, scheduling issues posed ongoing challenges in schools 
that did not operate on block schedules. The 90-minute READ 1/80 class reduced the time 
available for participating students to enroll in other classes since all other courses 
operated on a different schedule. 


Implementation of the Stanford Math Initiative in Year 4 reduced teachers’ ability to 
integrate literacy strategies into instruction because of the computerized and scripted 
nature of the supplemental program. 


New state curricular standards accelerated teaching the required curriculum so that 
teachers were compelled to teach two years’ worth of standards in one year. One principal 
felt this was particularly onerous for mathematics teachers in Year 4 and likely affected 
teachers’ MCLA implementation. 


Other changes, such as staff turnover and participation in new initiatives, affected MCLA 
implementation, especially among Cohort | schools who received the intervention during the 
2006 and 2007 school years. Despite these hurdles, one Cohort | school has intensified its focus 
on literacy integration across the curriculum, partly in response to district goals related to reading 
comprehension. Most principals valued the MCLA schoolwide initiative and saw evidence of 
continued implementation of MCLA strategies in their schools. 


Finally, survey information collected from teachers at the eight MSRP schools in May 2010 
reveals the extent to which teachers participated in professional development in Year 4 other 
than MCLA that may have complemented or “competed” with the intervention. (See Appendix 
K for the instrument.) Teachers were asked to think about the 2009-2010 school year when 
responding to questions about the following: 

* how many hours of professional development in specific topic areas they had received 
* how prepared they felt to engage in a set of 24 specific literacy activities 
* how often they had implemented the 24 literacy strategies 

Specifically, teachers were asked to respond to questions about time spent in professional 
development using a four-point likert scale, where 0 = “no hours spent,” 1 = “1 to 8 hours spent,” 
2 = “9 to 32 hours,” and 3 = “32 or more hours of professional development.” A total of 214 
(98.1%) teachers completed the Teacher Implementation of Strategies Questionnaire (TISQU) in 


May 2010: 102 (47.7%) respondents worked in Cohort | schools and 112 (52.3%) respondents 
were from Cohort 2 schools. Sixty-three (61.7%) of the Cohort 2 survey respondents had 
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participated in MCLA, representing 67 percent of all 94 Year 4 participants who had completed 
either semester of MCLA. 


Results from an independent ¢ test for equality of means show no cohort differences in Year 4 
with regard to the amount of time respondents reported participating in staff development in four 
areas: (1) their subject discipline; (2) state or district curricular standards; (3) class management; 
or (4) addressing the needs of students with disabilities. Cohort differences did emerge, however, 
with regard to the amount of time respondents spent in staff development in the past year in (1) 
new teaching methods (¢ = -2.43,df= 206, p < .05); (2) technology integration (¢ = -2.03, 
df = 211, p < .05); (3) student assessment (t = -2.09, df= 211, p < .05); (4) addressing English 
Language Learners (t = -2.64, df= 208, p < .05); and (5) literacy integration (#.-5.16, df= 211, 
p<.05). 


Table V-1 summarizes the means and corresponding standard deviations for responses about 
time spent in professional development that were significantly different by teacher cohort. Since 
responses ranged from “0 to 3”, a response of “2” indicates that a teacher reported receiving 
between 9 and 32 hours of professional development in a given topical area. As the table shows, 
the mean responses among Cohort 2 teachers were significantly higher than for Cohort | teachers 
in time spent in professional development in each of the five areas. 


Table V-1: Means and Standard Deviations for Survey Responses* to Items about Time 
Spent in Professional Development by Cohort (NV = 214), Year 4 


Standard 
Mean* Deviation 
New methods of teaching 
ss Conor TM 102) eft | O76 
Cohort 2 (N = 112) 1.60 0.78 
_Integration of educational technology 
= ee nc -ee o 2. 
Cohort 2 1.43 0.80 
ie Student performance assessment 
= Ce 2 
Cohort 2 1.59 0.80 
: Addressing the needs of ELL students 
~ ee A 
Cohort 2 1.03 0.91 
_Integrating literacy in the classroom 
. ne 9 a On 2 
Cohort 2 1.85 0.87 


Data source: RBS TISQU, May 2010 

* Significant at the p < .05 level 

T Based on the following scale: 0 = no time spent; 1 = spent 1 to 8 hours, 2 = spent 9 to 32 hours, and 
3 = spent 32 or more hours 
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Table V-2 shows the percentages of teachers who reported participating in each area of 
professional development for nine or more hours during the school year. Generally, respondents 
reported high levels of professional development participation in all topical areas measured. For 
example, as indicated in the table, 46.7 percent of the 214 survey respondents reported 
participating in nine or more hours of professional development in the subject area in which they 
primarily teach, and 46.6 percent of teachers spent at least nine or more hours in the past year 
engaged in learning new teaching methods. As expected, 62.5 percent of Cohort 2 respondents 
reported receiving nine or more hours of professional development in literacy integration. 


It should be noted that while teachers’ self-reported professional development participation 
signals a general trend, it is possible that teachers interpreted the professional development 
definitions differently when completing survey items. For example, in the first two years of this 
grant, many MCLA teachers underestimated the amount of time engaged in literacy training 
because they tended to define “professional development” in terms of in-service district events. 
In Year 4, respondents appear to have more accurately reported their MCLA participation. 


Table V-2: Percentages of Teachers Reporting that they Participated in Nine or More 
Hours of Professional Development in Past Year by MCLA Cohort, May 2010 (NV = 214)* 


Cohort 1 Cohort 2 


Schools Schools Total 
(N = 102) (N = 112) (N = 214) 
In-depth study in the subject area taught 42.2% 50.9% 46.7% 
New methods of teaching (e.g. cooperative learning) 40.0% 52.8% 46.6% 
State or district curriculum and performance 38.8% 51.8% 45.7% 
standards 
Integration of educational technology into the 29.7% 37.5% 33.8% 
classroom 
Student performance assessment (e.g. methods of 36.3% 47.7% 42.3% 


testing, applying results to modify instruction) 


Class management including student discipline 29.4% 31.2% 30.3% 


Addressing the needs of English language learners 


10) 19) 10) 
or students from diverse cultural backgrounds tea aie 20078 
Addressing the needs of students with disabilities 30.3% 30.3% 30.3% 
Integrating literacy in the classroom 30.7% 62.5% 47.4% 


Data source: RBS TISQU Survey, May 2010 
* Sample sizes for individual items varied slightly due to a minimal amount of missing data. 


The following subsections explore the extent to which the implementation of the teacher and 
principal professional development courses approximated the intended MCLA design through a 
review of Year 4 content and assignments. Next, the report presents the results of the analyses of 
teachers’ participation and retention rates, use of CRC materials, and collaboration with literacy 
coaches. These are followed by results from analyses of teacher implementation ratings 
provided by literacy coaches and classroom observations conducted by evaluators. Finally, the 
report presents implementation ratings evaluators calculated for each of the four participating 
Cohort 2 schools during Year 4. 
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MCLA Course Content 


Figure V-2 summarizes the topics addressed in the MCLA teacher course offered to Cohort 2 
teachers in Year 4. Course topics reflected the developers’ goal of promoting strategies aimed at 
building students’ vocabulary, fluency, and comprehension skills before, during, and after 
reading. Cohort 2 teachers were required to complete five classroom action plans (CAP) (three in 
the fall and two in the spring) and two poster presentations. Assignments in the fall 2009 
semester focused on using word maps and games to build vocabulary, student-generated 
questions for improving comprehension, and one-minute reading activities for increasing 
fluency. In spring 2010, developers reviewed before-, during-, and after-reading activities and 
addressed topics such as working with over-aged students, incorporating content literacy 
assessments into a lesson, and how to analyze student work. (See Appendix L for course syllabi 
and Appendix M for an example of a CAP). 


Figure V-2: MCLA Year 4 Course Topics 


Fall 2009 Spring 2010 
¢ Increasing Vocabulary Knowledge ¢ Introducing Anchor Activities 
* Teaching Tactic/Concept: Before, During, and After Reading | * Before, During, and After Reading 
Strategies Options for Fluency, Vocabulary, 
* Building Vocabulary: Semantic Features Analysis, Word and Comprehension 
Maps, Concept Maps, and Six Degrees of Separation Game_ | « Implementing a Content Literacy 
* Constructing a Pre- and Post-test for Whole Class Assessment 
Assessment Using Word Map * Homework and Parental 
* Optional Strategies: Classroom Blogging and Wiki Writing Involvement 
(with or without Technology) * Working with Over-Aged Students 
* Improving Comprehension: Marzano's Question Stems as * Analysis of Student Work for 
Pre/Post-test Assessments Planning Instruction 
* Previewing Text Using THIEVES and SEARCHES before 
Reading 
* Fix-up Strategies (During and After Reading) 
* Doing a Content Analysis to Identify Key Facts, Concepts, 
and Generalizations in a Unit of Study 
* Building Fluency: One-Minute Academic Reading, 
Neurological Impress, and “Scooping” 


Data source: MCLA syllabi, 2009-2010 school year 


In Year 4, the lead MCLA developer created a course template for all instructors to follow 
during each session of the evening course. Instructors for each content area (i.e., mathematics, 
social science, ELA, and science) adapted the template where necessary to address content- 
specific material. The templates included a rubric to guide the activity, a scoring sheet, an out-of- 
class assignment, and the CAP. (See Appendix N for an example of a fall 2009 instructor 
template on the topic of using Semantic Features Analysis, Word Maps, and Concept Maps to 
improve vocabulary.) 
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MCLA Course Participation 


In Year 4, MCLA professional development was expanded to include all full-time school 
personnel who provided instruction to students, including content and exploratory teachers, 
counselors who occasionally taught a class, and those teaching in special education self- 
contained classrooms. There was no official kick-off ceremony held in Year 4 as there had been 
previously, and professional development commenced on September 1, 2009. MCS files show 
that 156 teachers and school counselors were eligible for MCLA in fall 2009. As shown in Table 
V-3, the number of teachers completing the fall semester was 93, which represents 59.6 percent 
of the 156 eligible staff at Cohort 2 schools. Seventeen (16.0%) of the 93 fall completers 
discontinued MCLA after the fall semester; one new teacher completed only the spring semester. 
Overall, 94 teachers completed either Year 4 semester, and a total of 77 MCLA participants—or 
82.7 percent of all enrolled teachers—completed both fall and spring semesters. The percentage 
of eligible teachers at each school who enrolled in MCLA ranged from 44 percent at one school 
to 72.7 percent at another school in fall 2009; participation levels at these schools in the spring 
were 24 percent and 60 percent, respectively. 


Table V-3: Numbers and Percentages of MCLA Participants by School, Year 4 


Number 
Number of Completing Percentage of 
Eligible Number of MCLA both Eligible Teachers in Percentage of all 
School Teachers* Completers Semesters MCLA MCLA Completers 
Fall Spring Year 4 Fall Spring Fall Spring 
School A 55 40 33 33 72.7 60.0 43.1 42.3 
School C 25 11 6 6 44.0 24.0 11.8 7.7 
School H 48 27 24 23 56.3 50.0 29.0 30.8 
School L 28 15 15 15 53.5 53.5 16.1 19.2 
Total 156 93 78 77 59.6% 50.0% 100.0% | 100.0% 


Data sources: MCS district files and MCLA Attendance Sheets, 2009-2010 school year 
* All teachers were eligible to participate in MCLA, including content, exploratory, and special education teachers. 


In Year 4, developers offered a total of 42.5 hours of MCLA course-related professional 
development to participants. Staff provided nine fall evening sessions (20.25 hours) and nine 
spring semester classes (approximately 20.25 hours) in addition to an evening Laureate 
ceremony (2 hours) where teachers presented posters of their final work. As in previous years, 
evening sessions typically ran from 4:15 to 6:30 p.m. (Since teachers participated in 49 hours of 
course-related MCLA professional development during the previous year, developers delivered a 
total of 91.5 hours of classes over the two-year intervention period to Cohort 2 teachers. This 
total number of hours of service delivery excludes time spent with onsite literacy coaches.) 


Table V-4 summarizes the number and percentage of course participants by MCLA content 
area (ELA, mathematics, science, and social studies) and shows that the social studies course had 
the greatest percentage of MCLA participants, while science had the lowest percentage of 
participants (11.8%). 
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Table V-4: Numbers of Course Participants by MCLA Content Area, Year 4 


Percentage of all MCLA 
Number of MCLA completers Participants 
Content area Fall Spring Both Fall Spring Both 
ELA 28 23 23 30.1 29.5 29.9 
Mathematics 24 17 16 25.8 21.8 20.8 
Science 11 9 9 11.8 11.5 11.7 
Social studies 30 29 29 32.2 37.2 37.6 
Total 93 78 77 100% 100% 100% 


Data source: MCLA stipend lists, Year 4 


Table V-5 summarizes participants’ primary subject area taught in Year 4 and shows that 
42.6 percent of MCLA teachers taught a content area as their primary subject area, while another 
13.8 percent remaining teachers primarily taught special education or other classes, and 
6.3 percent taught reading. 


Table V-5: Percentage Distribution of MCLA Participants by Primary Subject Area 
Taught, Year 4 (NV = 94) 


Number of Percentage of All 
Participants MCLA Participants 
Content Area 42.6 


Reading 6 6.3 


Special Education 13 13.8 
Exploratory i 
= PS rN nc a 
a Computer/Technology fT 

Foreign Language 3 3.2 

Other (e.g., counselor) 16 17.1 

Unknown 8 8.5 


Data source: MCS District data files, 2010 


Table V-6 summarizes the number and percentage of MCLA completers who had high 
attendance (defined as having attended 80 percent or more of the evening professional 
development sessions) in either academic semester and for the overall school year. Results show 
that attendance among Cohort 2 participants was reasonably high: 55 (71.4%) of the 77 
participants who completed both semesters had, on average, an attendance of 80 percent of the 
sessions or more. Specifically, 20 (86.9%) of the 23 ELA attendees completing both semesters 
had an average attendance of at least 80 percent of the sessions; 11 (68.8%) of the 16 teachers in 
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the mathematics area had high attendance; and 24 (82.8%) of the 29 social studies completers 
had high attendance. None of the teachers attending science met the threshold for high 
attendance in Year 4. 


Table V-6: Percentages of MCLA Teachers with High Attendance* by Content Area, Year 


4 
Fall 2009 Spring 2010 Year 4 
Total Total Total High 
Completing Completing Completing | Attendance 
Content High Content High Both for Full 
Content Area Classes Attendance Classes Attendance Semesters Year! 
# # % # # % # # % 
ELA/READ 180 28 22 78.5 23 17 73.9 23 20 86.9 
Mathematics 25 16 64.0 17 9 52.9 16 11 68.8 
Science 11 0 0.0 9 0 0.0 9 0 0.0 
Social Studies 30 16 53.0 29 17 58.6 29 24 82.8 
Total 94 54 57.5 78 43 55.1 77 55 71.4 


Data source: Instructors’ MCLA attendance sheets 
*Defined as attending 2 80 percent of sessions 
™RBS averaged the attendance rates for those completing fall and spring semesters. 


MCLA Principal Fellowship Course Participation 


In Year 4, MCLA developers met with building principals and other school leaders to discuss 
issues related to school literacy and MCLA implementation. The class—comprised of 11 
individuals from four schools—met four times on the following dates: (1) September 14, 2009; 
(2) October 20, 2009; (3) November 16, 2010; and (4) February 1, 2010. Four (36.4%) 
participants had perfect attendance, four (36.4%) attended three times, two (18.1%) attended 
twice, and one individual (9.0%) attended one session. Three of the four Cohort 2 principals 
attended all four sessions, while one principal missed a session due to scheduling conflict. Three 
attendees worked as instructional support specialists/facilitators and the remaining four 
participants worked as vice principals. 


Information collected from principals through structured interviews suggests that each found 
the meetings professionally valuable and tended to characterize the experience as enhancing their 
professional knowledge and skills. Respondents particularly praised developers for providing 
opportunities to interact with other principals about best practices for implementing literacy 
strategies in their schools. Principals stated that MCLA facilitators encouraged them to conduct 
school walkthroughs with a focus on literacy, which they suggested strengthened their own 
observational techniques and sparked discussions with teachers about instructional approaches. 
They shared the following insights about their experiences in the leadership seminar: 


* Participation helped principals to support teachers more effectively by improving their 
understanding of what MCLA teachers were learning and what the expectations were for 
teachers’ implementation of literacy strategies. 
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* The research-based articles provided through the seminar were valuable for informing best 
practices in the school. 


* Respondents learned techniques for ensuring that teachers implement a range of different 
strategies to support student achievement. 


Three of the principals indicated that they expect to sustain the professional network 
established through the fellowship meetings. This assessment was based on the prior existence 
of a closely-knit group of middle school principals and the fact that the principals recognized the 
value of sharing with each other their experiences of successful practices in their schools. 


Curriculum Resource Center (CRC) Use 


In addition to funding the professional development course and literacy coaching services, 
the MSRP grant provided participants with a CRC that housed an array of reading materials and 
themed resources for use with their students. The repository included resources such as National 
Geographic leveled text thematic sets, readers’ theater sets, TIME Secondary Science series, and 
Hampton Brown’s Picture It! The CRC was operational on the first day of classes in Year 4 and 
all teachers, regardless of MCLA participation, were permitted to use the resources. 


One of four schools furnished evaluators with logs that indicated how many teachers 
borrowed CRC materials. In Year 4, 10 (25%) of the 40 MCLA participants at School A 
borrowed a total of 28 resources from the repository. At this school, one teacher checked out 11 
(39.2%) of the 28 resources, which were National Geographic Reading Expeditions sets for the 
physical sciences. Other teachers there borrowed materials such as Picture It!, TCM kits by the 
company Teacher Created Materials that contained readers’ theater scripts, and a themed set 
called Extreme Weather. 


Overall, ratings of teachers’ CRC use provided by the literacy coaches indicate low levels of 
use in Year 4. Of the 83 teachers for whom coaches assigned fidelity of implementation level 
ratings (described in greater detail momentarily), 34 (41%) participants did not use the CRC at 
all, 24 (or 28.9%) had low levels of use, 17 (20.5%) had medium levels of use, and seven (9.6%) 
teachers used the CRC to a high degree. According to the information provided by coaches, 
approximately 85.3 percent of MCLA participants from School A, 66.7 percent of participants 
from School C, 33.3 percent of participants from School H, and 38.5 percent of participants from 
School L used the CRC at least once during Year 4. These coach ratings were corroborated by 
the checkout logs that were provided by one school (School A). 


Literacy Coaching Support 


During the 2009-2010 school year, the team of six literacy coaches recorded daily tasks in 
logs that were designed jointly with RBS and the grant director. The coaching daily activity 
sheet (CDAL) included twelve categories of tasks that coaches might typically perform, such as 
conducting observations or conferencing with teachers. Evaluators entered and coded the 
information into Excel and SPSS databases (See the Appendix O for the instrument). Table V-7 
shows the number of CDALs submitted by each coach and the corresponding percentage of total 
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working days that the number of logs represents. The percentage is based on a 190-day year, or 
the number of days in the school year plus two weeks before the start of school during which 
coaches worked with other school staff members to prepare for the students’ return. Only logs 
containing specific tasks were counted; logs indicating leave time (i.e., sick, vacation, or 
personal time) were excluded from the analysis. In all, coaches submitted a total of 667 daily 
logs in Year 4. 


Six coaches provided support to teachers at the four Cohort 2 schools. Two onsite coaches 
worked in School A; one onsite coach worked in School C; a floating literacy coach divided her 
time among three schools (Schools C, H, and L); one onsite coach worked at School H; and a 
sixth coach (the grant director) stepped in to provide coaching support at School L following a 
coach’s resignation. The floating coach also provided support to teachers at School L. 


Table V-7: Numbers of Daily Logs Submitted by Coaches in Year 4 (N = 667) 


% of Workdays 
# Documented 

School A 

Onsite coach 1 150 78.9 

Onsite coach 2 132 69.5 
School C 

Onsite coach 124 65.3 

Floating coach* 13 6.8 
School H 

Onsite coach 151 79.5 

Floating coach* 59 31.1 
School L 

Coacht 10 5.3 

Floating coach* 28 14.7 


Data source: MCLA Literacy Coach CDALs, 2009-2010 
* Represents the same individual 
T The coach did not work full-time onsite 


Researchers entered 4,085 individual task items from the 667 CDALs submitted by coaches. 
Table V-8 summarizes the types of activities logged. Administrative tasks (V =1,188) accounted 
for 29.1 percent of the 4,085 tasks logged, followed by non-MCLA school-related tasks (19.1%), 
and activities related to training or meeting with teachers (17.5%). It is important to note that for 
every interaction “of substance” between coach and teacher, there are corresponding 
administrative tasks (such as delivering MCLA supplies or paperwork, scheduling meetings, 
maintaining the CRC, and photocopying MCLA-related materials). Professional development 
for the literacy coaches in Year 4 included, but was not limited to, the following: READ 180 
network meetings, an all-day session about teacher-created supplemental instructional materials, 
a technology training, and sessions with the MCLA developer to review classroom action plans. 
Approximately 19 percent of coaches’ tasks involved non-MCLA school-related tasks, such as 
assisting with administering the state test or helping with school security during arrival and 
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dismissal times. (This percentage increased from 16.5 percent of all tasks logged in the previous 
school year, which is not shown in the table.) The fourth-most commonly logged type of task 
was helping teachers prepare for class (6.7%), followed by performing MCLA-related school 
tasks, such as recruiting program participants (6.5%). Coaches logged fewer instances of tasks 
related to classroom observation (5.7%), in-class assistance (5.1%) (e.g., substituting for the 
main teacher) and performing tasks related to this evaluation (2.1%). Lastly, tasks related to the 
evening professional development course (1.2%), modeling lessons (0.3%), and videotaping 
teachers (0.2%) consumed a very small percentage of coaches’ overall tasks documented. 


Table V-8: Types of Coaching Tasks Recorded, Year 4 


Task Type Year 4 (N = 4,085) 
Frequency % 
Coach administrative task 1,188 29.1 
Non-MCLA school tasks 781 19.1 
Trained or met with teachers 715 17.5 
Helped teacher prepare for class 274 6.7 
MCLA-related school tasks 266 6.5 
Coach Professional Development 265 6.5 
Observed teacher 232 5.7 
Assisted teacher in other ways during class 210 5.1 
SR Evaluation tasks 87 2.1 
Related to teaching evening course 44 1.2 
Modeled lesson 14 0.3 
Videotaped 9 0.2 
Total 4,085 100 


Data source: MCLA Literacy Coach CDALs, 2009-2010 


Coaches’ READ 180 Tasks 


In Year 4, coaches described 233 tasks that were related to the READ 180 (targeted) 
intervention, accounting for 5.7 percent of the total 4,085 activities logged (not shown in table). 
The percentage of tasks devoted to READ 180 was lower than with Cohort | teachers: READ 180 
tasks comprised 12.5 percent of coaching tasks in Year | (or 447 of 1,804 overall tasks) and 
10.4 percent in Year 2 (or 600 of 5,791 total tasks logged). The proportion of tasks devoted to 
READ 180 in Year 4 was approximately the same as in Year 3, when READ 180 tasks consumed 
4.3 percent (or 215 of 4,085) overall tasks logged. READ 180 tasks included observing and 
providing feedback to READ /80 teachers, providing them with materials, and attending 
meetings and training sessions related to the program. 


Coach Availability 


RBS assessed coaching dosage and availability in Year 4 using three methods: (1) a review 
of entries in the coaching logs that referenced specific teachers’ names, (2) weekly surveys 
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administered during the fall semester, and (3) a survey administered at the end of the spring 
semester. 


First, RBS calculated the number of times each MCLA teacher appeared by name in the Year 
4 coaches’ logs and the number of times they were referenced in a substantive or meaningful 
way (e.g., working with the coach on lesson plans rather than on administrative tasks). It is 
important to note that since 2,979 (72.9%) of 4,085 tasks logged did not include a specific 
teacher’s individual name and evaluators could not locate an identification number for a handful 
of teachers in the dataset, the analysis underestimates the number of teachers whom coaches 
served. The proportion of tasks associated with a specific teacher represents only one-third of all 
tasks logged; however, analyzing entries that describe identified teacher participants nevertheless 
yields rich insight into the “dosage” of coaching provided. 


Table V-9summarizes the number of MCLA teachers by school who appeared substantively 
in the coaching logs 10 or more times, considered by RBS to represent adequate or “high” 
coaching dosage as part of the whole-school intervention. In addition, the table summarizes the 
median numbers and ranges of teachers whom coaches served in Year 4. The percentage of 
teachers receiving a high coaching dosage was highest in Schools C and A, at 63.6 percent and 
53.8 percent, respectively. One of these schools was large, while the other relatively small. 
While 22.2 percent of teachers at School H received a high coaching dosage, none of the teachers 
at school L received a high coaching dosage according to the coaches’ log entries and RBS 
criteria. In all, 36.2 percent of the 94 total Year 4 MCLA participants received high levels of 
coaching support according to the manner in which coaches described the support in coaching 
logs. Perhaps coaches focused on providing more intensive support to those individuals who 
they judged had a greater need for coaching support than other teachers. 


Readers should know that while coaches encouraged teachers to collaborate, the level of 
participation/involvement was the teachers’ prerogative; some teachers may not have sought 
coaching assistance and therefore received a low dosage of coaching support. Moreover, the 
coach’s record-keeping style or level of specificity in documenting teacher-level interactions 
affected the analysis because tasks described without teacher names in the logs were not included 
in the teacher-level frequencies reported. As a result, the number of participants reported to have 
received high levels of coaching is likely to be understated. Nevertheless, the mean number of 
times that identified teachers were documented in the CDALs was as high as 18 at school C and 
as low as 3 at school L. 
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Table V-9: Numbers and Percentages of MCLA Teachers with High* Coaching Dosage, 
Year 4 


MCLA Year 4 Number of Times Teachers Receiving 
Completers at | Teachers in | Teachers Were Recorded High Coaching 
School Coach Log by Name in Log Dosage* 
# # median range # % 
School A 40 39 12 2 to 23 21 53.8 
School C 11 11 13 2 to 44 7 63.6 
School H 28 27 9 1 to 41 6 22.2 
School L 15 15 2 0to9 0 0 
Total 94 92 34 36.2 


Data source: MCLA Literacy Coach CDALs, 2009-2010 
* Appearing 2 10 times in log in substantive activities with coach 


In fall 2009, RBS collaborated with MCLA instructors to administer a Weekly 
Implementation of Literacy Activities (WILA) survey that asked teacher participants to provide 
self-reports of information about a range of activities, including whether or not a teacher worked 
with the literacy coach in the past seven days. (The survey’s other foci will be reviewed 
momentarily.) All 93 (100%) MCLA participants in the fall completed at least one survey over 
the six-week period; however, most respondents completed five (32.3%), six (22.6%), or seven 
(11.8%) checklists over time. One-third (33.3%) of respondents completed four or fewer 
checklists. (See Appendix P for the instrument and full report that presents all results.) 


Seventy-seven (82.8%) of the fall MCLA participants completed checklists frequently 
enough to allow for a comparison of results at three different time points. RBS analyzed the 
responses of teachers who completed checklists at either weeks three or four (baseline), five or 
six (midterm), and weeks eight or nine (follow-up). Results show that the percentage of 
respondents reporting meeting with a literacy coach in the prior seven days increased over time: 
44.2 percent (34 of 77 teachers) had met with the literacy coach at baseline, compared with 
74 percent at midterm (57 teachers), and 75.3 percent (58 teachers) at follow-up. 


In spring 2010, RBS administered a final survey to teachers at the closing MCLA ceremony 
to gauge perceptions about various aspects of the program, such as the extent to which 
participating in MCLA helped respondents to integrate literacy into lessons more often than had 
they not participated. A total of 44 spring MCLA participants (57.1% of 77 MCLA completers) 
responded to the survey. Evaluators asked respondents how often they worked with an MCLA 
literacy coach during the school year using a five-point scale ranging from “never” to “every 
week” (see Appendix Q for the instrument.) The phrase “working with a coach” was defined as 
“having a meeting, discussed, or created a classroom action plan together, or collaborated in 
some other way.” 


Results shown in Table V-10 show that over three-quarters (77.3%) of respondents reported 
working often with the literacy coach during Year 4. Teachers self-reported information 
reinforces the finding that activity documented in coaching logs underestimates the full range of 
support that coaches provided. Additional information not shown in the table indicates that each 
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(100%) of the survey respondents who answered three additional questions about the coaches 
(N = 42) agreed that her coach’s advice was helpful and that the coach was willing to help the 
teacher as requested. 


Table V-10: Teachers Self-Reported Frequencies of Meeting with Coaches during School 
Year by School (N = 44) 


Sometimes or 
Every Week Often Rarely 
# % # % # % 
School A 2 9.5 18 85.7 1 4.8 
School C 0 0.0 11 100.0 2 0.0 
School H 4 23.5 2 64.7 0 11.8 
School L 0 0.0 3 75.0 1 25.0 
Totals 6 13.6 34 77.3 4 9.1 


Data source: RBS Participant Feedback Survey, spring 2010 


Classroom-Level MCLA Implementation 


In addition to capturing data about relationships with literacy coaches, the WILA surveys 
offer insight into teachers’ self-reported classroom-level implementation of MCLA strategies. 
Survey items were aligned with dimensions described in the IC Map and reflected key activities 
that MCLA developers identified as crucial to successful classroom literacy integration. (As 
indicated previously, see Appendix P for the WILA instrument and expanded summary report). 
The survey asked teachers if in the past seven days they had met with a literacy coach, received 
feedback from an administrator regarding literacy integration, or had used any specific literacy 
strategies they learned in MCLA. Respondents were also asked the extent to which they had 
assisted students during implementation of any of those strategies. Finally, each week, teachers 
were asked if they had engaged in any of the six activities, including assigning students to 
cooperative groups, pre-assessing students’ content knowledge, and meeting with grade-level 
colleagues during the school day to integrate literacy into lessons. 


Table V-11 summarizes the number and percentage of the 77 respondents who reported that 
they engaged in activities in the past seven days at the three time points. At baseline, 
respondents were most likely to report that they had put students into cooperative groups with 
assigned roles (72.7%), that they had informally assessed their students’ use of an MCLA 
strategy (67.5%), and that they had pre-assessed their students’ content knowledge (70.1%). 
Over time, baseline to follow-up increases were noted in the percentage of teachers who reported 
that they formally assessed strategy use (a change from 25.9% to 75.3%), met with literacy coach 
in school (a change from 44.2% to 75.3%), and received feedback from an administrator with 
regard to literacy instruction (a change from 14.3% to 24.7%). Receiving feedback from an 
administrator was the least frequent activity reported by teachers at all three-time points. 


RBS conducted an analysis of variance (ANOVA) to determine whether the average number 
of activities reported by each teacher (between zero and eight) was statistically different at the 
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three different time points. The test revealed no statistical difference in the average number of 
activities reported by the 77 teachers at the different time points (p< .05); however, results show 
that two of the eight activities were reported at significantly higher rates over time: teachers’ 
meeting with literacy coach (p <. 05) and preassessing the student content knowledge (p < .05). 


Table V-11: Numbers and Percentages of Respondents Reporting Engagement in Various 
Activities in the Past Week, Fall 2009 (N = 77) 


Baseline Midterm Follow-up 

# % # % # % 
Put Students into Cooperative Groups 56 72.7 50 64.9 50 64.9 
Informally Assessed Students’ Strategy Use 52 67.5 50 64.9 49 63.6 
Preassessed Students’ Content Knowledge 54 70.1 43 55.8 58 75.3 
Met with Literacy Coach 34 44.2 57 74.0 58 75.3 
Formally Assessed Students' Strategy Use 20 25.9 | 27.0 | 35.1 58 75.3 
Identified a Specific MCLA Strategy Used in Class 44 57.1 46 59.7 42 54.5 
Met with Grade-Level Colleagues 39 50.6 42 54.5 42 54.5 
Sanh Senin from Administrator about 14 143 18 23.4 19 247 


Data source: RBS WILA Survey, fall 2009 


Evaluator Observations 


In addition to collecting teachers’ self-reported perceptions about implementation, RBS 
observed 51 of the MCLA participants in Year 4 (representing 54.2 % of fall participants and 
66.2% of spring participants). More specifically, staff conducted observations of classes taught 
by MCLA teachers in October 2009 (N = 36 two-day class sessions and 6 one-day class 
sessions) and May 2010 (N = 32 one-day observations) to determine the extent to which teachers 
used strategies with students. In all, 74 lessons were observed in Year 4. Observers had 
completed one and a half days of training using videos featuring MCLA teachers implementing 
literacy strategies with students as well as a set of exercises that offered practice in note-taking, 
coding, and protocol completion. At both waves of data collection, observers conducted pre- and 
post-observation interviews with teachers to ascertain the context of lessons and the extent to 
which teachers perceived that lesson objectives had been met and to discuss next steps for class 
instruction. The observers used the RBS MSR Classroom Observation Protocol to document 
basic classroom characteristics (i.e., number of students, grade level, and content area) and the 
instructional and literacy strategies observed in ten-minute intervals. (See Appendix R for the 
instrument and annotated guide.) 


In addition to documenting the use of literacy strategies, observers were presented with a set 
of scenarios culled from descriptions of previously observed lessons to depict what observers 
would be likely to see. Evaluators demonstrated their ability to measure a lesson’s cognitive 
demand using a six-point scale based on Anderson and Krathwohl’s (2001) revised version of 
Bloom’s taxonomy of educational objectives. The scale includes the following levels, ranging 
from lowest to highest: (1) Remember; (2) Understand; (3) Apply; (4) Analyze; (5) Evaluate; and 
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(6) Create. To ensure that evaluators had a consistent understanding of the construct, significant 
time was spent reviewing and clarifying the rationale for assigning various ratings. Evaluators 
also designed and assigned ratings to additional two tasks and provided a rationale for ratings 
assigned to demonstrate their ability to adequately rate this dimension. 


RBS did not observe any lessons at a level of five or higher; observers noted components of a 
lesson that often required students to apply a procedure, such as synthesizing information from 
text into a graphic organizer. The observed cognitive demand level of lessons is elaborated upon 
in the results section below. 


Table V-12 summarizes the characteristics of the 51 teachers whom RBS observed in either 
the fall or spring of Year 4. As the table shows, eight in ten (82.4%) observed teachers were 
female, 70.6 percent were African American, and 60.8 percent had their professional teacher’s 
license (a figure that likely underestimates the proportion of teachers with full licensure due to 
missing survey data for this measure). An RBS cluster analysis of coaches’ assigned fidelity of 
implementation ratings found that 47.1 percent of observed teachers were rated as low-fidelity 
implementers, 27.5 percent were at medium fidelity levels, and one-quarter (25.5%) were at 
high-implementation-fidelity levels. Approximately 45 percent of the 51 observed teachers 
achieved a level of cognitive demand associated with application of concepts (level 3) and one- 
third (33.3%) reached a level where students were required only to understand concepts/material 
(level 2), while 19.6 percent of observed teachers achieved a level where students were expected 
to analyze information (level 4) presented during the lesson. Only one teacher (2.0%) failed to 
raise the cognitive demand level of the observed lesson beyond asking students to remember 
information (level 1). 


RBS included 42 fall classes and 30 spring classes in the final observation analysis after it 
deemed four observed special education inclusion classes ineligible for the study. Observers 
recorded a total of 633 strategies, or “episodes” of literacy across all observations. Table V-13 
summarizes the percentage of strategies employed by observed teachers and categorizes them 
into the three primary domains around which MCLA developers have designed the professional 
development model: strategies aiming to improve students’ vocabulary, comprehension, and 
fluency. Strategies that target more than one domain were coded as vocabulary/comprehension 
(i.e., from graphic organizers) or fluency/comprehension (e.g., reader’s theater activities). 
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Table V-12: Characteristics of Observed Teachers and Highest Cognitive Demand Level 


Measured, Year 4 (V=51) 


Number Percent Number Percent 
Gender Race/Ethnicity 
Female 42 82.4 Black, non-Hispanic 36 70.6 
Male 9 17.6 White, non-Hispanic 6 11.8 
Unknown 9 17.6 
Teaching Licensure 
Professional 31 60.8 Grade Level Observed 
Alternative 5 9.8 ELA 13 25.5 
Apprentice 4 7.8 Mathematics 10 19.6 
Interim/Other 3 5.9 Social Studies 10 19.6 
Unknown 8 15.7 Science 7 13.7 
Other 11 21.6 
Implementation Level 
Lowest 24 47.1 Education Level 
Medium 14 27.5 BA + 15 credits 10 19.6 
Highest 13 25.5 MA 13 25.5 
MA + 15 credits 17 33.3 
Highest Cognitive Demand Ed. Specialist 3 5.9 
Level Measured during Lesson Unknown 8 15.7 
1. Remember 1 2.0 
2. Understand 17 33.3 
3. Apply 23 45.1 
4. Analyze 10 19.6 


Data sources: RBS classroom observations, 2009-2010 and RBS TISQU survey, 2010 


Findings summarized in Table V-13 suggest that teachers who used multiple strategies 
tended to tap all three domains (vocabulary, fluency, and comprehension) with less emphasis on 
writing. More specifically, results from the analysis indicate that the percentage of vocabulary, 
fluency, and comprehension strategies used during observed lessons was relatively even: 28.1 
percent of strategies related to fluency, 29.9 percent related to comprehension, and 27.0 percent 
were a mix of vocabulary and comprehension (e.g., using graphic organizers). Observers coded 
only four (0.6%) instances of substantive writing aimed at integrating literacy in the lesson. 
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Table V-13: Percentages of Literacy Episodes Observed by Strategy Domain and Semester, 
Year 4 (NV = 633*) 


Strategy Domain Fall Spring Total 
% % % 
Vocabulary 14.4 13.3 14.1 
Fluency 26.0 32.8 28.1 
Comprehension 30.4 28.7 29.9 
Vocabulary/Comprehension* 28.1 24.6 27.0 
Fluency/Comprehension* 0.5 0.0 0.3 
Writing 0.7 0.5 0.6 
Total 100.0 100.0 100% 


Data source: RBS classroom observations, October 2009 and May 2010 

* 438 episodes were recorded across 42 fall observations, and 195 episodes were recorded across 30 spring 
observations 

T Includes the Frayer Model and other graphic organizers 

¥ Includes reader's theater. 


Although teachers employed a wide variety of literacy strategies, they most commonly used 
graphic organizers, read text aloud, engaged students in choral reading, or connected text to life 
in discussions during observed lessons. As Table V-14 shows, teachers used graphic organizers 
in over two-thirds (68.1%) of the observed lessons in Year 4. The least commonly used 
strategies were a visualization (NV = 1) technique and oral retelling (NV = 1), two strategies 
intended to improve students’ comprehension of text. Evaluators did not capture teachers using 
the question-answer-relationship (QAR) strategy on the day of observations, unlike during the 
previous school year. Moreover, although teachers were discouraged from using “popcorn 
reading” activities in which students read aloud solo, observers documented six classes using this 
approach in the fall (not shown in the table). 
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Table V-14: Numbers and Percentages of Lessons Observed with Various Literacy 
Strategies Used during Observation, Fall and Spring in Year 4 (NV = 72) 


Fall 2009 Spring 2010 
(N = 42 lessons) | (N = 30 lessons) Year 4 Totals 
# % # % # % 

Graphic organizer 30 71.4 19 33.3 49 68.1 
Teacher reads aloud 27 64.3 20 66.7 47 65.3 
Choral reading 17 40.5 13 43.3 30 41.7 
Connecting text to life 15 35.7 7 23.3 22 30.6 
Monitoring understanding of text 12 28.6 10 33.3 22 30.6 
Repeated reading 10 23.8 6 20.0 16 22.2 
Activating prior knowledge 10 23.8 6 20.0 16 22.2 
Previewing text 12 28.6 3 10.0 15 20.8 
Frayer Model 12 28.6 3 10.0 15 20.8 
Pre-teaching vocabulary 7 16.7 5 16.7 12 16.7 
Questioning for purpose 6 14.3 6 20.0 12 16.7 
Interactive word wall 7 16.7 4 13.3 11 15.3 
Glossary use 5 11.9 3 10.0 8 11.1 
Semantic features analysis 6 14.3 5 16.7 11 8.3 
Students generating questions 2 48 3 10.0 5 6.9 
Etymology 4 9.5 0 0.0 4 5.6 
Shared writing 3 7.1 1 3.3 4 5.6 
Context clue 3 7.1 1 3.3 4 5.6 
Journal or Blog Use 2 48 2 6.7 4 5.6 
Reader's theater 2 48 0 0.0 2 2.8 
Reflection 1 2.4 1 3.3 2 2.8 
Word sort 1 2.4 1 3.3 2 2.8 
Direct Instruction 1 2.4 1 3.3 2 2.8 
Think-Pair-Share 1 2.4 1 3.3 2 2.8 
Retelling 1 2.4 0 0.0 1 1.4 
Paired or buddy reading 0 0.0 1 3.3 1 1.4 
KWL 0 0.0 1 3.3 1 1.4 
Visualization 1 2.4 0 0.0 1 1.4 
Oral Retelling 1 2.4 0 0.0 1 1.4 


Data source: RBS Classroom Observation Protocol, 2009-2010 


Literacy Coach Ratings of Teacher Implementation 


In Year 4, MCLA literacy coaches maintained portfolios of teachers’ work and other 
documentation that included evidence sheets used by coaches during classroom observations, 
teachers’ instructional materials, student work artifacts, and lesson plans. The repository of 
information served as evidence upon which coaches drew to indicate a teacher’s level of MCLA 
implementation fidelity. At the end of the school year, RBS convened a working session with 
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five of the six literacy coaches for the purpose of assigning MCLA implementation ratings for 
individual Cohort 2 teachers on a subset of components using descriptions in the IC Map; the 
coaches were encouraged to cull evidence from the portfolios during the session to support their 
ratings. 


RBS selected six domains from the IC Map that reflected implementation fidelity areas that 
coaches were uniquely positioned to rate by virtue of frequent teacher interactions and 
observations. RBS facilitated coaches’ assignment of teacher ratings using the adapted rubric 
included in Appendix S, which described variations in implementation from the optimal level of 
MCLA strategy implementation, as designated by developers, to a minimal level of 
implementation. The six domains reflect the extent to which coaches observed teachers: (1) 
introducing strategies and describing its purpose when used; (2) consistently modeling the use of 
a strategy; (3) providing multiple guided practice activities using a variety of texts; (4) providing 
opportunities for students’ independent practice of strategies; (5) differentiating instruction based 
on analysis of progress monitoring; and (6) revisiting previously introduced literacy strategies 
and applying them to new material. 


The literacy coaches assigned fidelity ratings for 83 of 94 teachers with whom they worked 
during the 2009-2010 school year. Table V-15 summarizes the number and percentage of 
teachers rated at various levels of implementation for each of the six dimensions after two years 
of MCLA participation. Implementation levels ranged from low to high, with the number of 
levels varying dependent upon the component. Results show that coaches’ ratings of individual 
teachers were distributed widely across implementation domains; however, a cluster analysis 
showed that coaches rated over half (53%) of the 83 teachers as low-fidelity implementers, while 
they rated 26.5 percent as medium-fidelity implementers and 20.5 percent as high-fidelity 
implementers. Findings suggest that the IC Map is an effective framework for distinguishing 
among different levels of fidelity. 


Table V-15: Numbers and Percentages of Teachers Rated at Various Implementation 
Levels by Literacy Coaches by School in Year 4 (NV = 83) 


Numbers of Teachers 
Rated During 2009- Percentages of Teachers by 


2010 Implementation Level 
Low Medium High 
School A 34 41.2 29.4 29.4 
School C 9 55.6 33.3 11.1 
School H 27 51.9 29.6 18.5 
School L 13 53.8 30.7 15.4 
All Schools 83 53.0 26.5 20.5 


Data source: RBS Literacy Coach IC Map rating tool, 2010 


Table V-16 summarizes the percentage of teachers with ratings by school and specific IC 
map domain, such as the fidelity with which coaches felt teachers differentiated instruction or 
introduced and revisited literacy strategies. Results suggest a trend in which teachers from 
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School A tended to earn higher implementation fidelity ratings than teachers from other schools. 
Generally, it seems that teachers at two schools struggled most with differentiating instruction, 
and one coach rated a majority (85%) of School H teachers as low-fidelity in this area. 


Table V-16: Percentages of Teachers Rated at Various Implementation Levels by Literacy 
Coaches in Year 4 (N = 83) 


All 
School A School C School H School L Teachers 
(n = 34) (n= 9) (n = 27) (n= 13) (N = 83) 
Introduce Strategy 
lowest (%) 2.9 22.2 25.9 30.8 16.9 
medium (%) 47.1 44.4 40.7 38.5 43.4 
highest (%) 50.0 33.3 33.3 30.8 39.8 
Model Strategy 
lowest (%) 0.0 11.1 37.0 23.1 16.9 
medium (%) 23.5 33.3 25.9 38.5 27.7 
med/higher (2.5) 2.9 0.0 0.0 0.0 1.2 
higher (%) 29.4 55.6 25.9 23.1 30.1 
highest (%) 44.1 0.0 11.1 15.4 24.1 
Use Guided Practice 
lowest (%) 11.8 0.0 29.6 38.5 20.5 
medium (%) 29.4 55.6 29.6 23.1 31.3 
higher (%) 32.4 33.3 22.2 23.1 27.7 
highest (%) 26.5 11.1 18.5 15.4 20.5 
Encourage Independent Use of Strategies 
lowest (%) 14.7 33.3 33.3 30.8 25.3 
medium (%) 50.0 44.4 40.7 46.2 45.8 
highest (%) 35.3 22.2 25.9 23.1 28.9 
Differentiate Instruction 
lowest (%) 32.4 11.1 85.2 46.2 49.4 
medium (%) 41.2 55.6 11.1 23.1 30.1 
highest (%) 26.5 33.3 3.7 30.8 20.5 
Revisit Strategies 
lowest (%) 20.6 11.1 48.1 38.5 31.3 
medium (%) 29.4 44.4 18.5 23.1 26.5 
med/higher (2.5) 2.9 0.0 3.7 0.0 2.4 
higher (%) 14.7 33.3 14.8 38.5 20.5 
highest (%) 32.4 11.1 14.8 0.0 19.3 


Data source: RBS coaching rubric adapted from the MSRP IC Map, Year 4 


Summary of Implementation Findings for the Whole-School Intervention 


In Year 4 of the MSRP, developers provided 42.5 hours of out-of-school professional 
development to teacher participants to implement five lessons that integrated specific literacy 
practices before, during, and after reading as well as two poster presentations of exploratory 
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research findings to colleagues. Literacy coaches helped teachers complete the assignments 
through feedback provided in debriefing conferences, classroom observations, and a wide range 
of other general support activities. In addition to coaching assistance, teachers were encouraged 
to use materials and resources from an on-site curriculum library maintained by the literacy 
coaches. 


RBS tracked attendance at the MCLA evening classes to determine individual and 
schoolwide program participation in the four schools receiving the intervention in Year 4. 
Approximately 82 percent (NV = 77) of the 94 fall-enrolled teachers completed the yearlong 
professional development intervention. Teachers who attended and completed the courses 
tended to have high levels of course participation: 55 (71.4%) of the 77 teachers who completed 
both semesters attended 80 percent or more of classes; however, only 58.5 percent of all 
originally enrolled teachers achieved high course participation. 


Although course attendance was generally high among completers, enrollment in MCLA 
across the four schools varied widely, as in previous years: nearly three-quarters (72.7%) of 
eligible teachers participated in one school, compared with 44 percent of eligible teachers in 
another school in fall 2009. 


RBS assigned an implementation rating to each school using a formula that takes into 
account teachers’ course attendance, participation with literacy coaches, and the number of 
eligible teachers who opted not to participate in the program. All MCLA-eligible staff members 
in the school were assigned one of four numerical ratings depending on how many MCLA 
professional development sessions they attended in fall 2009 and spring 2010. Teachers’ 
attendance rates in the fall and spring were averaged for an overall attendance rate; for example, 
a teacher with 100 percent fall course attendance who dropped the course before the spring 
semester began received a “0” for spring and an average of 50 percent attendance overall. 
Eligible teachers who did not participate in MCLA received an attendance rate of “0.” The 
ratings are as follows: teachers who attended 25 percent or fewer of the sessions were given a 
“1,” those attending between 26 and 50 percent of the professional development offered were 
assigned a “2,” teachers participating in between 51 and 75 percent of the professional 
development offered received a “3,” and those who attended 76 to 100 percent of the 
professional development offered were given a rating of “4.” 


Other implementation ratings were assigned to the four MCLA schools, including a coaching 
dosage score, principal involvement rating, use of materials score, and teacher implementation 
score. The percentage of MCLA teachers with high coaching dosage was determined by the 
percentage of teachers who worked with the coaches 10 or more times during the school year in 
a substantive way according to data provided in weekly coaching logs. A principal involvement 
score of “4” was assigned to each school since three of four principals attended all fellowship 
classes and two key MCLA events and one principal attended all but one fellowship session. 
The use of materials rating is determined from a teacher-level use rating assigned by the literacy 
coaches and follows the same scale that was used to rate course participation scores (and 
includes non-MCLA participants). The teacher implementation score was derived from 
individual-level ratings of teacher implementation assigned by coaches that were aggregated to 
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the school level and based upon mean scores between “1” and “4.” Finally, the school’s 
implementation rating is a composite score based on the previous measures. 


Once the above ratings were tallied, RBS calculated an average score for each MCLA school 
and assigned it one of four corresponding schoolwide implementation ratings: 


1 = minimal program implementation 
1.1 to 2 = low implementation 

2.1 to 3 = medium implementation 
3.1 to 4 = high implementation 


Table V-17 summarizes these implementation ratings as well as the number and percentage 
of participants in the intervention by school for Year 4. Results show a low to medium level of 
implementation at each of the four Striving Readers schools, ranging from 2.54 at School A to 
1.91 at School L. As shown, the percentage of eligible teachers completing both semesters of 
MCLA in Year 4 was 60 percent at School A, 24 percent at School C, 54.2 percent at School H, 
and 53.5 percent at School L. RBS calculated the professional development score (“PD score for 
participants”) for each school by first by assigning a teacher-level attendance score to MCLA 
participants (using the method previously described where attendance of 75% or more of the 
sessions resulted in a score of 4) and then calculated the mean attendance score for all MCLA 
participants in the school. A separate PD score was calculated that included non-MCLA 
participants who were assigned a “0” score for MCLA attendance. 


RBS calculated the aggregated high coaching dosage score by first assigning a score of “1” 
to all teachers in the school who were documented 10 or more times in coaching logs. RBS 
assigned a score of “0” to all teachers (including non-MCLA participants) who did not work with 
the coach 10 more times. RBS then calculated a mean between 0 and | for each school, which 
represented the percentage of all teachers at that school who received a high dosage of coaching 
support. For example, of the 55 eligible teachers at School A, 22 (40%) teachers received a high 
coaching dosage (as indicated by the coaching logs). Finally, RBS assigned an aggregate school 
high coaching dosage score of “2” to represent the 40 percent of teachers falling into the overall 
typology of participation where “1” = “25 percent or fewer,” “2 = between 25 percent and 50 
percent,” “3 = between 51 and 75 percent,” and “4 = between 76 percent and 100.” At School C, 
28 percent of the 25 eligible teachers received a high dosage of coaching support, and therefore 
earned an aggregate school score of “1.” The scores for Schools H and L were “1” and “1” 
respectively, because of the relatively large numbers of non-MCLA participants at their schools. 


The aggregated score for coaches’ school-level implementation ratings were calculated as 
follows: coaches provided a total of 83 ratings for teachers (34 from school A; 9 from School C; 
27 from School H; and 13 from School L). Teachers received a rating of 1 (low), 2 (medium) or 
3 (high). RBS calculated the mean rating for all teachers in the school with implementation 
ratings. For example, in School A, the mean rating for the 34 teachers was 1.89. 


As previously mentioned, principal involvement was deemed high in all four schools, since 
attendance at 75 percent of the sessions or higher indicated high attendance. RBS calculated the 
“materials use” rating by analyzing teacher-level scores that were assigned by coaches and 
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calculating the mean score for each school (which again, includes non-MCLA participants). 
RBS assigned two of the schools a score of “2” which indicates that between 25 percent and 50 
percent of the teachers had a usage level above “low use” and two of the schools earned a score 
of “1,” which indicates the schools’ use of CRC materials was low. 


Table V-17: Schoolwide MCLA Participation and Implementation Rankings, Year 4 


School A School C School H School L 
Number of Eligible Teachers in Fall 2009 55 25 48 28 
Percent of Eligible Teachers Participating in 72.7 44.0 56.3 53.5 
MCLA 
Nee! Completing MCLA, Fall Semester 40 14 27 15 
(N = 93) 
Number Completing MCLA, Spring 
Semester (N = 78) ae $ a is 
Number (%) of Teachers Completing both 6 6 6 6 
Semesters (N = 77) 33 (60%) 6 (24%) 23 (54.2%) | 15 (53.5%) 
PD Score for Participants* 3 1 3 3 
PD Score including Nonparticipants 2.79 1.00 2.02 2.00 
Coaching High Dosage Score’ 2 (42.3 %) 2 (28%) 1 (12.7%) 1 (0%) 
Coach’s implementation Rating 1.89 1.33 1.55 1.54 
(Aggregated) 
Principal Involvement Rating 4 4 4 4 
Materials Use Rating’ 2 2 1 1 
School s Implementation Rating (includes 2.54 211 211 1.91 
nonparticipants) 


Data sources: MCLA attendance records, CDALs, RBS Literacy Coach IC Map Rating Tool 
*1=< 25% had high attendance, 2 = 26%—50% had high attendance, 3 = 51%—75% had high attendance, and 


4 => than 75% had high attendance 


T Score is based on the percentage of all teachers (as of fall 2009) who received high levels of coaching. Aggregate 
score includes non-MCLA participants who received no coaching where 1 = < 25% of teacher had high levels of 
coaching, 2= 26%—50 % had high levels of coaching, 3 = 51%—75% had high levels of coaching, and 4 = > than 
75% had high levels of coaching. 

* Based on N = 83 teachers for whom coaches assigned ratings. RBS calculated the mean score based upon 
teacher-level scores where coaches provided ratings of 1 = low, 2 = medium, and 3 = high fidelity of 
implementation. 

5 Materials use is determined using coaches’ scores for participants, where 1 = low and 2 = medium use. 
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Figure V-3: Characteristics of Year 4 RBS Data-Collection Methods 


Date 
Data Collection Method and Topic Conducted Sample size* 
Surveys 
Follow-up characteristics and content Noe 4 (On Sp dlye OF e.2 Sonor: 
P May 2010 administrators in all 8 Striving Readers 
knowledge—all content teachers 
schools) 
N = 93 (100%) MCLA participants 
completed at least one survey. A total of 
WILA Survey Fall 2009 77 (82.8%) of the MCLA teachers 
completed surveys at baseline, mid-term, 
and at follow-up. 
Feedback Survey (formerly entitled ; N = 44 (56.4% of the 78 spring semester 
Spring 2010 
the pre-focus group survey) completers) 
Interviews 
Striving Readers School Principals May 2010 N = 8 (100%) 
Literacy coaches May 2010 N =6 (100%) 
Observations 
October 2010 N = 15 (93.8%) 
READ 1780 cl 
mere May 2010 N = 15 (93.8%) 
October 2009 N = 81 (87.1% of 93 MCLA teachers) 
MCLA Cl 
seal May 2010 N = 31 (39.7% of 78 MCLA teachers) 
Student assessment 
Baseline ITBS* Fall 2009 N = 1,350 (sixth graders only) 
Follow-up ITBS* Spring 2010 N = 3,907 (all students) 
Secondary Data 
MCLA attendance rosters, 
READ 180meeting attendance sheets Bee leevavabie Gate 
Coaching calendar and log entries Year 4 N =6 coaches (100%) 
TCAP Spring 2010 Unknown until data are released 


*Where possible, response rates are provided in parentheses. 
TThese numbers include all students who sat for the test; therefore, they do not match N sizes included elsewhere in 
this report. 
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Vi. Quasi-Experimental Estimations of the Impacts of 
the Whole-School Intervention: Years 3 and 4 


During Years | and 2, the whole-school intervention was implemented in Cohort 1 schools, 
and the analysis used teachers from Cohort 2 schools as research controls; at the end of Year 2, 
the intervention ended in Cohort | schools. During Years 3 and 4, the intervention was 
implemented in Cohort 2 schools. The experimental phase of the whole-school intervention has 
ended, so no experimental impacts are presented in this report. However, exploratory analyses 
were carried out in an attempt to estimate any improvements in student reading achievement in 
the Cohort 2 schools at the end of Year 4 that might be attributable to the whole-school 
intervention. 


The quasi-experimental designs used compare the performance of students enrolled in the 
Cohort 2 schools at the end of Year 4 to the performance of students enrolled in the Cohort 2 
schools at the end of Year 2. The strength of this design is that each school serves as its own 
“control.” The weakness is that most of the students enrolled at the end of Year 4 are different 
from those enrolled at the end of Year 2. (The only students who were enrolled in both periods 
are the eighth-grade students in Year 4 who were in sixth grade in Year 2.) As in the impact 
analyses of MCLA in Years 1 and 2, the students who were included in the analytical samples 
were those who had been enrolled in one of the four Cohort 2 schools for a majority of 
instructional days in Years 2 or 4 for sixth-grade students and in Years | and 2 or 3 and 4 for 
seventh- and eighth-grade students. Independent sample t-tests were conducted to compare 
performance of baseline measures between Year 2 and year 4 cohorts for each grade level as well 
as for the combined seventh- and eighth-grade sample. Statistically significant mean differences 
were detected between sixth-grade samples only. None of the mean differences at baseline were 
more than 25 percent of the pooled sample standard deviation. Baseline performance was used 
as a covariate in the specified impact models to adjust for these differences. 


The ITBS Total Reading, Comprehension, and Vocabulary NCE scores obtained at the end 
of Years 2 and 4 were the dependent variables.”” The evaluation year (Year 2 or Year 4) was the 
“treatment” variable used to represent the difference between student achievement before and 
after the whole-school intervention. The analytical models used to estimate this treatment effect 
included three types of covariates: the appropriate baseline score for each dependent variable, 
several student demographic characteristics (gender, whether a student was eligible for free or 
reduced-price lunch, English language learner status, and whether a student was African- 
American), and three dummy variables representing the differences among the four schools. The 
appropriate ITBS administrations for each year and grade are indicated in Table VI-1. 


°° TCAP scores were not available until January 2011. Results of these exploratory analyses for TCAP scores will 
be included in the revisions to this draft report. 
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Table VI-1: ITBS Administrations Providing NCE Scores for Students in Each Evaluation 
Year and Grade 


Evaluation Year Student ITBS Pretest ITBS Posttest 
Grade Date Date 
Year 2 6 Fall 2007 Spring 2008 
7 Fall 2006 Spring 2008 
8 Fall 2006 Spring 2008 
Year 4 6 Fall 2009 Spring 2010 
7 Fall 2008 Spring 2010 
8 Spring 2008 Spring 2010 


Before beginning analyses to estimate the treatment effect, of MCLA, researchers compared 
the baseline ITBS scores of similar groups of students in Year 2 and Year 4. As shown in Table 
VI-2, these comparison tests show the following fairly small but statistically significant 
differences for sixth-grade students (only): -2.7 [¢(1117) = 2.89, p = .004] on the ITBS 
Vocabulary, -2.8 [#(1104) = 3.15, p = .002] on the ITBS Comprehension, and -2.9 [t(1097) = 
3.30, p = .001] on the ITBS Total Reading. 


Table VI-2: MCLA Baseline Comparability on ITBS NCE Scores in Year 4 Compared to 
Year 2—Sixth-Grade Students 


Means Degrees of t- Signif. 
Test score Year 2 Year 4 Freedom value Level 
ITBS 33.90 31.00 4097 3.30 0.001 
Total Reading (556)* (543) : , 
ITBS 36.00 33.20 
Comprehension (558) (548) 1104 3.15 0.002 
ITBS 33.20 30.50 
1117 2.89 0.004 
Vocabulary (573) (546) 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 
schools 

* Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline fall 
2009 administration and the spring 2010 administration. 
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Table VI-3: MCLA Baseline Comparability on ITBS NCE Scores in Year 4 Compared to 
Year 2—Seventh-Grade Students 


Means Degrees of t- Signif. 
Test score Year 2 Year 4 Freedom value Level 
ITBS 33.60 33.70 
ITBS 35.80 35.10 
Comprehension (467) (425) 890 0.74 0.462 
we —— sche 852 -1.28 0.203 
Vocabulary (470) (384) , ; 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 


schools 
* Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline fall 


2008 administration and the spring 2010 administration. 


Table VI-4: MCLA Baseline Comparability on ITBS NCE Scores in Year 4 Compared to 
Year 2—Eighth-Grade Students 


Means Degrees of t- Signif. 

Test score Year 2 Year 4 Freedom value Level 

ITBS 34.30 33.00 844 1.23 0.219 
Total Reading (508)* (338) 

ITBS 35.20 34.90 

eorniercroh aa Aa) 850 0.21 0.831 

ITBS aed eee 862 1.84 0.066 
Vocabulary (521) (343) 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 


schools 
* Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline fall 


2008 administration and the spring 2010 administration. 


Table VI-5: MCLA Baseline Comparability on ITBS NCE Scores in Year 4 Compared to 
Year 2—Seventh-Grade and Eighth-Grade Students 


Means Degrees of t- Signif. 
Test score Year 2 Year 4 Freedom value Level 
ITBS 35.50 35.00 
Total Reading (980)" (764) 1742 0.63 0.531 
TES soe eaue 1742 0.63 0.531 
Comprehension (980) (764) , ; 
ITBS 33.40 33.00 
1716 0.51 0.608 
Vocabulary (991) (727) 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 


schools 
* Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline fall 


2008 administration and the spring 2010 administration. 


Three analytic approaches were employed to estimate the evaluation year treatment effect: 
Ordinary Least Squares (OLS) regression analyses, analyses of simulated outcomes in both 


Memphis Striving Readers/RBS: Evaluation and Impact Report June 17, 2011—Page 104 of 114 


directions, and difference-in-differences analyses. The results of the first two approaches are 
presented below. The difference-in-differences results are included as Appendix T. The intent is 
to compare the results of these different approaches. Of particular interest is the comparison of 
the OLS regression and the simulated-outcomes approaches. The number of students included in 
all of these analyses is described in Table VI-6. 


Table VI-6: Number of Students in Each Evaluation Year and Grade 


Number of 
Evaluation Year Grade Students 
Year 2 6 779 
7 611 
8 676 
Year 4 6 708 
7 527 
8 585 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 
schools 


Variables Included in Both Analyses 


The regression model and the simulated-outcomes analysis used for each of the three ITBS 
scores included the posttest NCE score as the dependent variable; three dichotomous (0,1) 
variables representing the Cohort 2 schools; a dichotomous variable representing the evaluation 
year (the “treatment’); four dichotomous variables representing student gender, eligibility for 
free or reduced-price lunch, English language learner status, and whether a student was A frican- 
American; and the baseline score for the appropriate dependent variable. 


OLS Regression Analysis Results 


All variables were entered into the analysis in order to facilitate the comparison of the 
different approaches. Tables VI-7 through VI-10 present the results for each grade separately 
and for seventh and eighth grades together. The results for sixth grade represent the impact of 
one year of enrollment in a school receiving the whole-school intervention. The results for 
seventh and eighth grade represent the impact of two years of enrollment. 
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Table VI-7: MCLA “Impact” on ITBS NCE Scores in Year 4 Compared to Year 2—Sixth- 


Grade Students 


Unadjusted Means Adjusted Means Est. Effect Signif. 

Test Score Year 2 Year 4 Year 2 Year 4 Impact Size Level 
es Reading ae ae 28.6 28.9 0.3 0.02 0.624 
a nee ha ae 31.4 31.0 -0.4 -0.02 | 0.607 
ee ak oe 28.6 28.8 0.2 0.01 0.738 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 


schools 


* Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline fall 


2009 administration and the spring 2010 administration. 


Table VI-8: MCLA “Impact” on ITBS NCE Scores in Year 4 Compared to Year 2— 
Seventh-Grade Students 


Unadjusted Means Adjusted Means Est. Effect | Signif. 

Test Score Year 2 Year 4 Year 2 Year 4 Impact Size Level 
a Reading a ay 31.9 31.9 0.0 0.00 0.970 
Reker een aah a 34.1 33.8 -0.3 -0.02 0.741 
ees aaa) el 31.3 31.5 0.2 0.01 0.835 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 


schools 


* Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline fall 


2008 administration and the spring 2010 administration. 


Table VI-9: MCLA “Impact” on ITBS NCE Scores in Year 4 Compared to Year 2— 
Eighth-Grade Students 


Unadjusted Means Adjusted Means Est. Effect Signif. 

Test Score Year 2 Year 4 Year 2 Year 4 Impact Size Level 
ae Reading can ae 31.0 33.3 2.3 0.15 0.005 
ep eaier ier ees 33.1 34.0 0.9 0.05 0.331 
eos ek ai 31.0 34.1 3.1 0.19 0.001 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 


schools 


* Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline fall 


2008 administration and the spring 2010 administration. 
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Table VI-10: MCLA “Impact” on ITBS NCE Scores in Year 4 Compared to Year 2— 


Seventh-Grade and Eighth-Grade Students 


Unadjusted Means Adjusted Means Est. Effect Signif. 

Test Score Year 2 Year 4 Year 2 Year 4 Impact Size Level 
es Reading ae a 31.5 32.4 0.9 0.05 0.135 
esrprekanees iso) an 33.6 33.9 0.3 0.02 0.618 
ee (ean a 31.3 32.6 1.3 0.08 0.056 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 
schools 

* Numbers in parentheses are the numbers of students in each group having valid test scores from the baseline fall 
2008 administration and the spring 2010 administration. 


The estimated impact of the whole-school intervention was not significant (p0.05) in any 
grade except eighth. For eighth-grade students, the Year 4 results were significantly better than 
in Year 2 for total reading and vocabulary. The effect sizes approached 0.20. When the eighth- 
grade students were combined with the seventh-grade students, the effect sizes were reduced and 
the effects were not significant, despite the increased sample size. 


Analysis of Simulated Outcomes of MCLA in Year 4 


This procedure examined actual outcomes from Years 2 and 4 and simulated outcomes from 
the same years. The simulated outcomes are estimates of scores that Year 2 students would have 
achieved if they had experienced MCLA and scores that Year 4 students would have achieved if 
they had not experienced MCLA. The differences between these simulated outcomes and the 
actual scores from Year 2 and Year 4 are calculated. Finally, to reduce the amount of bias from 
possible idiosyncratic variations between the years, the differences are averaged to produce a 
robust simulation of the overall effect of MCLA. 


The covariates used are the same as those listed in the “Variables Included in Both Analyses” 
subsection above. In this subsection, the following symbols are used to indicate the actual and 
simulated outcomes (NCEs of ITBS scores): 


Y; = Actual outcome for Year 4 students 


Y, = Simulated counterfactual outcome for Year 4 students (if Year 4 students had not 
experienced MCLA) 


S; = Actual outcome for Year 2 students 


S; = Simulated counterfactual outcome for Year 2 students (if Year 2 students had 
experienced MCLA) 


To create the least-biased estimate of the overall treatment effect of MCLA, the evaluators 
followed a simulation procedure that combined the cross-validation technique (Picard & Cook, 
1984; Geisser, 1993) and a post-matching analytical approach to estimate the average treatment 
effect introduced by Ho, Imai, King and Stuart (2007; 2011, pp. 18-19). Ho, et al’s (2007; 2011) 
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causal inference approach was carried out using the R package Zelig (Imai, King, & Lau, 2008). 
The evaluators carried out the simulation analyses in SAS.”! 


Analytic Procedure 


Step 1: Evaluators fit a multiple linear regression model to the Cohort 2 schools’ Year 2 student 
data. Next, the evaluators multiplied each of the coefficients estimated from the regression 
model by the observed Year 4 values of the corresponding covariates. The sum of the products 
of regression coefficients and the observed covariate values gave the estimated counterfactual 
outcome for the Year 4 students (Y)). The following is an example of the regression equation 
that was used to estimate the counterfactual outcome, with Yrepresenting the estimate of a 
student’s ITBS Total Reading score: 


Y, = By+ B, (pretest ITBS Tot)+ >» (Female)+ 3 (Free or Reduced Lunch) + 4 (ELL) + 
Bs (AfriAmerican) + Bs (School A) + 87 (School B) + Bs (School C) 


Step 2: Using the Cohort 2 schools’ Year 4 data, evaluators constructed a multiple linear 
regression model with the same set of independent variables as those in the model built in Step 1. 
Next, the resulting regression coefficients were multiplied by covariate values from the Year 2 
students. This provided an estimate of the counterfactual outcomes for the Year 2 students (5). 
Since the Year 2 students did not recetve MCLA, the estimated outcomes are predictions of the 
ITBS test scores that students would have achieved if they had received MCLA. 


Step 3. Evaluators calculated the treatment effect of MCLA for the Year 4 sample by subtracting 
the simulated outcome from the actual outcome, and calculated the Year 2 treatment effect by 
subtracting the actual outcome from the simulated outcome. The resulting differences are two 
simulated estimates of the treatment effect of MCLA: 


Y;- Y, = estimate of simulated treatment effect of MCLA in Year 4 
S, - S; = estimate of simulated treatment effect of MCLA in Year 2 


Step 4. To estimate the overall effect of MCLA, evaluators computed the weighted average of 
treatment effects for the Year 4 and Year 2 samples with respective sample size as the weight. 
Using the sample-weighted average of treatment effects reduced the bias of estimates due to 
idiosyncratic nature of data in either sample. 


Figure VI-1 is an illustration of Step 1 of this estimation of the least biased average treatment 
effect. Evaluators plotted the observed scores of Year 4 students on the y-axis and the simulated 
counterfactual scores of Year 4 students on the x-axis, as shown in Figure VI-1. The evaluators 


*! The difference between the approach used in the present study and Ho et al.'s (2007; 2011) approach and regular 
cross-validation technique are: (1) the evaluators didn't apply the simulation based on matched data and hence had 
uneven treatment and comparison sample sizes. (2) the evaluators calculated the sample-weighted average of 
treatment effects for the treated (Year 4 cohort) and the comparison (Year 2 cohort), so the larger sample would 
contribute more weight to the estimates of the overall MCLA effect. (3) The regular cross-validation usually starts 
with splitting the sample into halves randomly while the two subsamples, i.e., the Year 2 and Year 4 cohorts, were 
defined in advance in the present approach. (4) The regular cross-validation approach uses stepwise selection in 
building the multiple regression model for one subsample, but the evaluators forced all the covariates in the model 
so that the variable selection would be consistent with that based on OLS regression. 
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calculated the difference between observed and simulated values for each of the students in the 
Year 4 sample, or Y,-Y;, denoted by e in the figure. Each “e” in the figure is a random example 
of the difference between observed and simulated values, and the brackets indicate the vertical 
distance between the data point (the “e) and the diagonal, or the effect estimate for that student. 
Although this is not shown in the figure, evaluators also calculated the difference between 
simulated and observed values for students in the Year 2 sample. The average of these 
differences is the simulated estimate of the average treatment effect of MCLA. 


Figure VI-1: Example of Scatterplot of Observed Outcome Scores vs. Simulated 
Counterfactual Scores. 
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Table VI-7 summarizes the findings of treatment effects for the Year 4 and Year 2 students 
as well as the overall treatment effects for sixth-grade students, seventh-grade students, eighth- 
grade students, and seventh- and eighth-grade students combined. Multiple regression analyses 
were conducted separately for Year 4 and Year 2 samples. For the Year 4 sample, the first 
column, “Obs’d (T)’, represents the observed outcome for the students. Because Year 4 sample 
received MCLA, the observed outcome is considered the treatment outcome, denoted by the “T” 
in parentheses. The next column, labeled “Est’d (C),” represents the predicted counterfactuals 
estimated using the regression estimates derived from Year 2 data. The “Diff” indicates the 
difference between the observed and predicted outcomes. Column “SD (C)” represents the 
standard deviation of the predicted ITBS post-test scores. 
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A paired-samples t-test was conducted to test the difference between each pair of observed 
and model-simulated outcomes. The associated p value is marked in bold if the difference is 
statistically significant at the .05 level. By dividing the difference by the standard deviation of 
the estimated posttest scores, the evaluators also obtained the standardized effect size (ES). A 
similar procedure was also applied to Year 2 data. Because Year 2 students did not receive 
MCLA, the observed ITBS posttest scores are considered the comparison outcome, as shown in 
column “Obs’d (C).” The column labeled “Est’d (T)” represents the predicted counterfactuals 
simulated using Year 4 data. 


As shown by Table VI-11, there are eight statistically significant findings, of which the 
eighth-grade students’ ITBS Vocabulary in Year 4 yielded the highest treatment effect size at 
.33, while the seventh- and eighth-grade students combined in Year 2 had the lowest effect at 
.06. It is noteworthy that the analyses revealed statistically significant simulated impacts on both 
the treated and comparison students for the ITBS Total Reading and Vocabulary scores of the 
students in eighth grade as well as the ITBS Vocabulary scores of the students in seventh and 
eighth grades combined. Therefore, the weighted average treatment effects for these three sets of 
outcomes are more robust than the others. 


In particular, two years of MCLA seems to have had an overall treatment effect of .24 
standard deviations on the eighth-grade students’ ITBS Vocabulary scores, an overall treatment 
effect of .17 standard deviations on the eighth-grade students’ ITBS Total Reading scores, and an 
overall treatment effect of .09 standard deviations on the seventh- and eighth-grade students’ 
ITBS Vocabulary scores. 


Conclusion 


In conclusion, there is a high level of agreement between the results produced by the OLS 
regression method and the approach of using simulated outcomes. The significance level and 
magnitude of effects on eighth-grade students’ ITBS Vocabulary and Total Reading scores are 
very close to the estimates derived from the OLS regression analyses: the effects on both 
vocabulary and total reading scores are about .20. The two approaches also agree on the 
magnitude of effect of MCLA on seventh- and eighth-grade students’ vocabulary scores with 
ES = .08 for OLS and ES = .09 for the simulated-outcomes approach, although the estimated 
effect using OLS is marginally significant (p = .056). The two approaches presented above (as 
well as the difference-in-difference methods presented in Appendix T) produced highly similar 
results despite the differences in modeling and estimating the MCLA impact. This fact provides 
the researchers with strong evidence that the detectable significant impacts of MCLA treatment 
on the students included in these analytic samples is likely to be accurate. These results seem to 
indicate that MCLA is a professional development program worthy of additional study. 
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Table VI-11: Results of Simulated Outcome Analysis of MCLA Impact in Year 4 


Outcome Variable 


ITBS Total Reading 


Year 4 


Obs’d Estd sD 
7 (C) ON Dif (C) ES ft 


27.77 27.41 543 0.36 11.41 0.03 . 


_Year2 
Obs’d Est'd © sD 
(Tt) (C) ON Diff (C) ES ip 


29.83 29.63 556 0.20 15.17 0.01. 


Avg. TX 
effect 


0.02 


ITBS Comprehension 


31.77 32.34 556 -0.58 15.64 -0.04 


-0.03 


Grade 6 


ITBS Vocabulary 
ITBS Total Reading 


29.98 30.27 543 -0.29 10.94 -0.03 . 
28.03 27.72 543 0.32 9.71 0.03 . 


31.62, 31.50, 384 0.12, 11.15 0.01, 


29.54 29.19 556 0.34 14.77 0.02. 
32.81 32.07 464 0.75 14.97 0.05. 


0.03 
0.03 


ITBS Comprehension 


Grade 7 


33.77 33.71 384 0.05 10.68 0.00 . 


35.97 34.38 464 159 14.86 0.11. 


0.06 


ITBS Vocabulary 
ITBS Total Reading 


31.64 31.12 384, 0.05 


30.62 338. 


0.52 9.90. 


33.01. 2.39 10.89 0.22 <.001 


31.79 31.31 464 0.48 16.39 0.03 .463 
33.37 31.23 508 2.13 16.10 0.13 <.001 


0.04 
0.17 


ITBS Comprehension 


Grade 8 


34.23 33.35 338 0.87 9.81 0.09 .203 


33.67 33.06 508 0.62 15.90 0.04 .287 


0.06 


ITBS Vocabulary 
ITBS Total Reading 


33.63 30.61 338 3.03 9.23 0.33 <.001 


32.27 31.40 722 0.87 10.67 0.08 .036 


34.53 31.67 508 2.86 15.95 0.18 <.001 
32.37 31.63 972 0.74 15.57 0.05 .053 


0.24 


0.06 


ITBS Comprehension 


33.98 33.67 722 0.31 9.86 0.03 .488 


34.13 33.69 972 0.44 15.42 0.03  .267 


0.03 


Grades 7&8 


ITBS Vocabulary 


32.57 31.32 722 1.25 8.99 0.14 .008 


32.47 31.50 972 0.97 16.16 0.06 .029 


Data sources: Years 2 and 4 demographic, pretest and posttest NCE scores of ITBS for students in Cohort 2 schools 
* The p values were derived from the paired-samples t-tests of the difference between observed and estimated outcomes. 


0.09 
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Study Conclusions 


Four years of analyses of data related to READ 180 indicate that implementation varied 
widely across the eight Striving Readers schools’ classrooms but has improved each year. 
Percentages of classrooms that were rated as adequate with regard to overall implementation 
were lowest in Year | (slightly above 40%) and in Year 4 reached levels of more than 
80 percent. Despite these ratings indicating adequate implementation, impact estimates resulting 
from this four-year randomized control trial fail to provide evidence to suggest that READ 180 
produces improved student performance beyond that of “business-as-usual” instruction provided 
by MCS to address the needs of adolescent struggling readers. Furthermore, the lack of 
significant findings has been consistent when analyzed across time or when ITT samples have 
been combined to increase power and regardless of whether HLM or propensity score matching 
techniques are used. 


Regarding the schoolwide intervention, Year 4 results indicate low to medium levels of 
MCLA implementation at each of the four Striving Readers schools. As one part of RBS’ 
implementation study, a cluster analysis of teachers’ ratings related to classroom implementation 
of strategies based upon the MCLA Innovation Configuration (IC) Map and assigned by coaches 
was conducted. Profiles describing distinct practices of teachers in each group resulted in over 
half (53%) of the 83 teachers characterized as low-fidelity implementers, 26.5 percent as 
medium-fidelity implementers, and 20.5 percent as high-fidelity implementers. Findings suggest 
that the IC Map is a sufficiently sensitive framework to support empirical fidelity studies. 


Year 4 represents completion of the second and final round of implementing MCLA, and 
ordinary least squares regression (OLS) and a quasi-experimental simulation analysis were 
carried out in an attempt to estimate improved achievement among students in the Cohort 2 
schools that might be attributable to the whole-school intervention. The estimated impact of 
MCLA using either approach indicated similar findings. Specifically, significant improvements 
were detected for eighth-grade students’ ITBS Vocabulary and Total Reading Scores, with effect 
sizes using either method estimated at around 0.20. Similar estimates were produced when 
calculating the magnitude of program effects on a combined sample of seventh and eighth grade 
students, although the OLS findings were not statistically significant. Taken together, 
particularly given the current climate with a federally driven focus on teacher training and 
teacher quality, these findings suggest that the revised MCLA model implemented in the Cohort 
2 schools during Years 3 and 4 may be a promising professional development approach worthy 
of additional study. 
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Appendix ES: Year 5 Plans for Dissemination 


The following is a list of Year 5 efforts to disseminate findings from the Memphis Striving 
Readers Project. Only Year 5 events, defined as those occurring between March 1, 2010, and 
December 31, 2011, are included. 


Presentations and Papers (completed) 


Coffey, D. (2010, March). Mid-South Striving Readers Project Targeted Intervention: 
Secondary Analyses. Panel presentation at the Society for Research of Educational 
Effectiveness Conference, Washington, DC. 


Feighan, K., & Harris, R. (2010, April). Evaluating a Multi-Year, Federally Funded Educational 
Initiative: Lessons from a Successful School District—-Evaluator Partnership. Presentation at 
the Annual Eastern Evaluation and Research Society Conference, Absecon, NJ. 


Feighan, K., & Heeren, E. (2010, April). The Data are in: Four Years of Coaching 
Implementation & Analysis. Presentation at the Second Annual International Literacy 
Coaching Suumit. Corpus Christi, TX. 


Feighan, K., Heeren, E., & Feldman, J. (2010, May). Exploring the Relationship between 
Teachers’ Literacy Strategy Use and Adolescent Achievement. Paper presented at the 
American Educational Research Association Annual Meeting, Denver, CO. 


Feighan, K., and Heeren, E., (2010). “She was My Backbone: Measuring Coaching Work and 
Its Impact,” in Literacy Coaching: Research & Practice edited by Cassidy, Garrett, and 
Sailors. Center for Educational Development, Evaluation, and Research (CEDER); Texas A 
& M University, Corpus Christi, College of Education. 


Heeren, E., Heaston, R., Pollan, C., Hall, C., and Feighan, K. (2010, April). Literacy Coaching: 
Measuring the Impact in Urban Schools. Presented at the International Reading Association 
Annual Meeting in Chicago. 


Heeren, E., and Feighan, K. (2010, June). Literacy Strategies in MCS Teachers’ Toolkit: 
Lessons from the Field. Presented at the Forum for Innovative Leadership 2010 Annual 
Meeting in Memphis, TN. 


Feldman, J.M. (2010, November). Empirically Establishing Implementation Fidelity of an 


Intervention for Struggling Reader. Presentation at the American Evaluation Association 
Annual Conference, San Antonio, TX. 


Feldman, J.M. (2010, November). Striving Readers: Results from the Mid-South. Panel 
presentation at the American Evaluation Association Annual Conference, San Antonio, TX. 


Rui, N., Bai, H., & Coffey, D. (2010, November). A Comparison of Genetic Matching and 
Propensity Score Matching Methods for Covariate Adjustment in a Reading Intervention 
Program Evaluation. Paper presentation at the American Evaluation Association Annual 
Conference, San Antonio, TX. 


Appendix ES. MSRP Evaluation Report, Y4 June 17, 2011. Page 2 of 3 


Feldman, J.M., Coffey, D., Rui, N. & Schenck, A. (2011, March). Striving for Balance: Using 
Rigorous Methods to Challenge Insignificant Findings, Presentation at the Society for 
Research of Educational Effectiveness Annual Conference, Washington, DC. Feighan, K., 
Heeren, E., and Feldman, J. (2011). The Correlates of Middle School Teachers’ 
Professional Development Implementation, Instructional Beliefs, and Student Reading 
Achievement, paper commissioned by the US Dept of Education, retrievable from 
http://slcp.ed.gov/other-resources/slcp-sponsored-publications/issue-papers/. 


Heeren, E., Pollock, C., Chitman, L., Feighan, K., and Feldman, J., (2011, April). Collaborating 
with Literacy Coaches to Measure Implementation Fidelity, Symposium presented at the 
Third International Literacy Coaching Summit, Philadelphia, PA. Rui, N., Feldman, J.M., 
Bai, H., Schenck, A. (2011, April). An Empirical Comparison of Genetic Matching and 
Propensity Score Matching for Covariate Adjustment, Roundtable presentation at the 
American Education Research Association Annual Conference, New Orleans, LA. 


Feldman, J.M., Feighan, K.A., and Heeren, E. (2011, April). Aiming High: Implementation 
Fidelity, Cognitive Demand, and Struggling Readers’ Literacy Outcomes, Roundtable paper 
presentation at the American Education Research Association Annual Conference, New 
Orleans, LA. 


Feldman, J.M., Coffey, D., Rui, N. & Schenck, A. (2011, April). Striving for Meaning: Using 
Rigorous Methods to Challenge Insignificant Findings. Presentation at the American 
Education Research Association Annual Conference, New Orleans, LA. 


Heeren, E., and Feldman, J. (2011, May). Today’s Readers, Tomorrow’s Leaders: Literacy 
Strategies that Impact Student Performance, Presentation at the International Reading 
Association 56" Annual Convention, Orlando, FL. 


Cooter, R., Cardenes-Lopez, E. Cooter, K., Magpuri-Lavell, T., and Ogle, D. (2011, May). 
Breakthroughs in Academic Vocabulary Instruction with Urban Adolescent Learners: Results 
from Striving Readers Research, Presentation at the International Reading Association 56" 
Annual Convention, Orlando, FL. 


Pollan, C., Heeren E., Feighan, K. (2011, June). Literacy Coaches in the Demonstration 
Classroom: A Sustainability Model, Presentation at the Forum for Innovative Leadership, 
Memphis, TN. 


Presentations (accepted) 


Heeren, E., Feldman, J., & Feighan, K. (2011, November). Today’s Readers, Tomorrow’s 
Leaders, Classroom demonstration accepted for presentation at the National Council of 
Teachers of English Conference, Chicago, IL. 


Heeren, E., Feldman, J., & Feighan, K. (2011, November). Today’s Readers, Tomorrow’s 
Leaders: Literacy Strategies for Urban Middle Schools. Accepted for presentation at the 
National Middle School Association Conference, Louisville, KY. 
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Presentations (submitted) 


Rui, N., Feldman, J. & Schenck, A. (2011, November). A Simulation- Based Causal Inference 
Approach Using Cross-Sectional Data. Paper submitted for presentation at the 32"! Annual 
Association for Public Policy Analysis and Management Conference, Boston, MA. 


Feldman, J., Coffey, D., Rui, N. & Schenck, A. (2011, November). Striving for Balance: The 
Value of Publishing Rigorous Studies with Insignificant Findings. Proposed for presentation 
at the American Evaluation Association Annual Conference, Anaheim, CA. 


Journal Publications (planned) 


A refined version of the AERA Aiming High paper exploring relationships between 
cognitive demand, fidelity of implementation and student outcomes following completion of a 
two-year intensive literacy intervention for middle school content area teachers. 


A methodological comparison of results using OLS, Simulation, and Differences of 
Difference Approaches to estimate two-year impact of an intensive literacy intervention for 
middle school content area teachers on striving adolescent readers’ achievement in reading. 


Findings from four Years of ITT analyses of READ 180 and an exploratory analysis of 
differential effects on struggling adolescent readers of various reading levels. 


Measuring Fidelity of Implementation: Opportunities, Tradeoffs, and Land Mines. 


Other Publications in progress 


“Improving Adolescent Literacy: Strategies and Professional Development Based on Work 
with Striving Readers.” Led by Elizabeth Heeren, a team of authors representing Memphis City 
Schools and the University of Memphis is preparing a monograph based on the Striving Readers 
project experience. The content will include fluency strategies, vocabulary strategies, 
comprehension strategies, sample lesson plans, and how to build school capacity to support 
reading achievement. Presently in the draft stage, the development schedule calls for completion 
and conversion of the manuscript into a commercial-quality book to be published in December 
2011. Keith Kershner (RBS) is leading the editing and production process. RBS Publications 
will disseminate the resulting book until at least 2016. 
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Memphis, Tennessee 


A. Students’ Use of Literacy Strategies When Reading Content-Relevant Texts in Core Content Classes or for Homework 


A.1. Checklist of MCLA literacy strategies that students should have learned to use (by skill domains of reading and strategy lead) 


Reading Skill Teacher or 
Code Domain Name of Strategy Student 
S.1 C Retelling (verbally, graphic organizers, written) Ss 
8.2 C Student-generated questions S 
8.3 C Student-led Question Answer Relationships S 
S.4 C&V Graphic Organizers S 
S.5 C&V Semantic Maps S 
S.6 C&V Student-led Thinking Maps (flow chart, double bubble, etc.) S 
S.7 F Choral reading (group/whole class) S 
8.8 F Paired reading (partners) Ss 
S.9 Vv Frayer Model S 
S.10 Vv Semantic Feature Analysis Ss 
S.11 Vv Student-led Word Sorts (open- and closed sorts) S 
T.O1 ALL Before, During, After T 
T.02 ALL Choice (teachers and learners) T 
T.03 ALL Combining Strategies- "Layering over time" T 
T.04 ALL Cooperative Learning T 
T.05 ALL Explicit, Direct Instruction (Gradual Release of Responsibility) T 
T.06 ALL Instructional Conversations (CREDE) T 
T.07 ALL Joint Productive Activity (CREDE) T 
T.08 ALL Motivating Learners T 
T.09 ALL Small Group Instruction T 
T.10 ALL Use of leveled, supplemental materials (e.g., National Geographic) T 
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Innovation Configuration Map for the Memphis Content Literacy Academy (MCLA) Program 


Memphis City Schools / University of Memphis 


Memphis, Tennessee 


Reading Skill Teacher or 
Code Domain Name of Strategy Student 
T.11 C Bloom's Taxonomy/Stem Questions T 
T.12 C Expository Text Structures T 
T.13 C METS T 
T.14 C ReQuest T 
T.15 C Teacher-led Question Answer Relationships (QARs) T 
T.16 C Think-Pair-Share T 
T.17 C Writing Organizer/Framework (K. Cooter) T 
T.18 C&V Thinking Maps (flow chart, double bubble, etc.) T 
T.19 F Choral Reading (Antiphonal, Unison, Echo) T 
T.20 F Radio Reading T 
T.21 F Repeated Readings T 
T.22 F Scooping T 
T.23 Vv Explicit Vocabulary Instruction T 
T.24 Vv Pre-Instruction of Vocabulary T 
T.25 Vv Pronunciation Review T 
T.26 Vv Word maps T 
T.27 Vv Teacher-led Word Sorts (open- and closed sorts) T 
T.28 Vv Word Walls (Academic) T 
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Innovation Configuration Map for the Memphis Content Literacy Academy (MCLA) Program 
Memphis City Schools / University of Memphis 


Memphis, Tennessee 


A.2. Students’ Use of Literacy Strategies (in collaborative/cooperative activity with peers, independent use) 


a b c d e f 
A.2.a. Degree of students’ Students can self- Students demonstrate Students can use Students are aware of the | Students engage 
independent use of MCLA select a strategy and | independent use of the strategies with peers strategy, can somewhat in text-based 


strategies: Students exhibit, 
when appropriate, independent 
and integrated use of multiple 
strategies. 


use it independently. 


strategy (without teacher 
or peer assistance) when 
the teacher tells them to 
use a Strategy. 


strategy. 


(cooperative or 
collaborative use) when 
teacher tells them to use a 


or scaffolding. 


use it but not without 
some teacher assistance 


work without the 
use of strategies. 


a 


b 


c 


d 


e 


A.2.b. Student roles and 
behaviors during cooperative 
learning activities: Students 
have assigned roles, carry out 
those roles, and exhibit 
behaviors consistent with class 
norms for cooperative learning 
activities (e.g., observing 
equity of voice, listening for 
understanding, offering 
positive feedback, appreciating 
contributions of others, etc.). 


Students have assigned roles 
but do not carry out roles. 
Students do exhibit behaviors 
consistent with class norms 
for cooperative learning 
activities (e.g., observing 
equity of voice, listening for 
understanding, offering 
positive feedback, 
appreciating contributions of 
others, etc.). 


Students are grouped for tasks 


but do not have assigned roles. 


Students exhibit some 
behaviors consistent with 
class norms for cooperative 
learning. 


Students do not have assigned 
roles and do not exhibit 
behaviors consistent with class 
norms for cooperative learning 
activities. 


There is no evidence that 
students are grouped in 
cooperative learning 
activities. Students work 
alone. 
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Innovation Configuration Map for the Memphis Content Literacy Academy (MCLA) Program 


Memphis City Schools / University of Memphis 


Memphis, Tennessee 


B. Core Content Teachers Plan and Implement Lessons Integrating Literacy Strategies with Instruction on Core Content 


B.1. Using information from assessments for revising instructional plans and developing plans for interventions 


a 


b 


c 


d 


B.1.a. Use of assessments for content 
learning: Teacher uses before, during, 
and after (end-of-unit) instructional 
assessments designed to provide 
information on the extent to which 
specific content-related learning 
objectives are being achieved. 


Teacher uses before and after 
(end-of-unit) instructional 
assessments designed to 
provide information on the 
extent to which specific 
content-related learning 
objectives are being achieved. 


Teacher uses summative 
(end-of-unit) instructional 
assessments to provide 
information on the extent to 
which specific content- 
related learning objectives 
are being achieved. 


Teacher uses summative 
(end-of-unit) 
instructional assessments 
to assign grades. 


B.1.b. Use of assessments for learning 
content literacy strategies: Teacher 
uses before, during, and after (end-of- 
unit) instructional assessments 
designed to provide information on the 
extent to which content literacy 
strategies are being learned and used 
appropriately. 


Teacher uses before and after 
(end-of-unit) instructional 
assessments designed to 
provide information on the 
extent to which specific content 
literacy strategies are being 
learned and used appropriately. 


Teacher uses summative 
(end-of-unit) instructional 
assessments to provide 
information on the extent to 
which specific content 
literacy strategies are being 
learned and used 
appropriately. 


Teacher uses 
instructional assessments 
but not to provide 
information on the extent 
to which specific content 
literacy strategies are 
being learned and used 
appropriately. 


B.1.c. Revision of instructional plans: 
Teacher uses information from 
instructional assessments of student 
progress with respect to specific 
content objectives to help him or her 
make revisions to instructional plans. 
In addition, teacher uses information 
from instructional assessments of 
students’ independent use of content 
literacy strategies to help him or her 
make revisions to instructional plans. 


Teacher uses information from 
instructional assessments of 
student progress with respect to 
specific content objectives to 
help him or her make revisions 
to instructional plans. In 
addition, teacher uses 
observations of students’ 
appropriate use of content 
literacy strategies to help him 
or her make revisions to 
instructional plans. 


Teacher uses information 
from formative assessments 
with respect to specific 
objectives to help him or her 
make revisions to 
instructional plans. 

In addition, teacher 
comparisons of students’ 
reading level with the content 
text(s) are used to help him 
or her make revisions to 
instructional plans. 


Teacher uses information 
from a single content 
pre-assessment of 
specific objectives to 
help him or her make 
revisions to instructional 
plans. The teacher does 
revise instructional plans 
based on students’ use of 
content literacy 
strategies. 


Teacher uses his/her 
knowledge of content 
objectives to plan 
instruction and does not 
revise instructional 
plans. 
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B.1. Using information from assessments for revising instructional plans and developing plans for interventions 


a b c d e 
B.1.d. Design interventions for Teacher uses information from | Teacher uses information Teacher uses information | Teacher designs any 
struggling students: Teacher uses instructional assessments to from instructional from a single supplementary 


information from instructional 
assessments to plan supplementary 
instruction for most struggling 
students. 


plan supplementary instruction 
for some struggling students. 


assessments to plan 
supplementary instruction for 
a few struggling students. 


instructional pre- 
assessment to plan 
supplementary 
instruction for a few 
struggling students. 


instruction provided to 
the whole class and 
does not use 
information from 
instructional 
assessments or design 
interventions to meet 
the needs of individual 
struggling students. 


B.2. Providing explicit, direct instruction, and practice (daily instruction, teacher modeling, guided practice) 


a 


b 


c 


B.2.a. Introduction of strategies: Teacher 
(1) names the strategy and (2) describes 
the purpose of the strategy and when it is 
to be used. Teacher activates students’ 
background knowledge and experiences 
to help them understand the strategy. 


Teacher mentions the 
strategy but does not 
provide students with a full 
description of the purpose of 
the strategy and when it is to 
be used. 


Teacher provides content 
instruction only. 


B.2.b. Teacher modeling: In providing 
explicit and direct instruction, teacher 
consistently models initial use of the 
strategies (e.g., think-alouds, 
questioning). 


In providing explicit and 
direct instruction, teacher 
occasionally models initial 
use of the strategies. 


Teacher makes passing 
reference to the strategy 
with no modeling provided. 


Teacher provides content 
instruction only. 


B.2.c. Guided practice: In providing 
explicit and direct instruction, teacher 
consistently provides multiple guided 
practice activities using a variety of texts. 
Students receive relevant feedback with 
respect to their use of specific strategies. 


In providing explicit and 
direct instruction, teacher 
occasionally involves 
students in guided practice 
activities and provides 
general feedback. 


In providing instruction, 
teacher involves students in 
follow-up activities without 
feedback. 


Teacher provides 
instruction without guided 
practice. 
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B.2. Providing explicit, direct instruction, and practice (daily instruction, teacher modeling, guided practice) 


a b c d e 
B.2.d. Independent use: Teacher provides | Teacher provides Teacher uses continual 
opportunities for students’ independent opportunities for students’ teacher-directed whole-class 
practice and monitors students’ progress | independent practice but instruction to guide 
applying strategies to assess additional does not monitor students’ students’ strategy 
learner needs. progress. application. 
B.2.e. Differentiated instruction: Teacher | Teacher differentiates Teacher relies primarily on 
differentiates instruction based on instruction but does not use | whole-group instruction. 
analysis of progress monitoring (e.g., data to flexibly group 
small groups, use of technology, students. 


reteaching, use of curriculum resource 
center materials) 


B.2.f. Revisiting of strategies: Teacher occasionally Teacher makes passing Teacher introduces each 
Teacher consistently revisits previously revisits previously reference to previously strategy once but does not 
introduced literacy strategies as introduced literacy taught strategies without revisit when new material 
opportunities to apply them to new strategies as opportunities to | providing opportunities for is presented. 

material. apply them to new material. | students to apply those 


strategies to new material. 


B.3. Objectives of instructional plans (core content knowledge and skills, literacy strategies) 


a b c d e 
B.3.a. Objectives in terms of core | Teacher’s instructional Teacher’s instructional plans Teacher’s instructional Teacher’s instructional plans 
content standards’ learning plans are linked to describe what core content plans are general and/or are not available. 
objectives, knowledge, and skills: | content standards and knowledge and skills will be non-specific. 
Teacher’s instructional plans are related to prior learning. | worked on during the lessons. 
linked to content learning Plan has vague reference to 
objectives and related to prior content standards. 
learning and students’ real life 
applications. 
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Memphis, Tennessee 


B.3. Objectives of instructional plans (core content knowledge and skills, literacy strategies) 


a 


b 


c 


d 


e 


B.3.b. Objectives in terms of 
literacy strategies: Teacher’s 
instructional plans include literacy 
strategies appropriate to the 
learning task and sequencing of 
the lessons. Lesson plans include 
activities that guide students 
toward independent application of 
literacy strategies. 


Teacher’s instructional 
plans match appropriate 
literacy strategies 
matched to learner needs 
to assist learners in 
acquiring core content 
knowledge. 


Teacher’s instructional plans 
match appropriate literacy 
strategies to assist learners in 
acquiring core content 
knowledge but without a match 
to learner needs. 


Teacher’s instructional 
plans reference the use of 
literacy strategies randomly 
(not embedded in use of 
text, appropriate to the 
sequence of the lesson—use 
of strategy for “use of 
strategy” instead of 
matching learning needs and 
sequencing to appropriate 
use of strategy). 


Teacher’s instructional plans 

make no reference to literacy 

strategies and only target core 
content. 


B.4. Using different instructional materials 


a 


b 


d 


e 


B.4.a. Use of adopted 
textbook: 


At least 3 days per week (and 
in all class periods, teacher 
helps students read and learn 
content from the adopted 
textbook. 


1-2 days per week in at least 
75% of class periods, 
teacher helps students read 
and learn content from the 
adopted textbook. 


1-2 days per week in at least 
50% of class periods, teacher 
help students read and learn 
content from the adopted 
textbook. 


Occasionally (at least monthly 
in at least 1 class period), 
teacher helps students read and 
learn content from the adopted 
textbook. 


Teacher does not help 
students read and learn 
content from the adopted 
textbook. Teacher expects 
students to read and learn 
content from the adopted 
textbook without help. 


B.4.b. Use of MCLA 
supplementary materials: At 


1-2 days per week in at least 


least 3 days per week (and in 75% of class periods, 
all class periods, teacher helps | teacher helps students select 
students select MCLA MCLA materials 


materials appropriate for their 
reading level, and read and 
learn content from those 
materials related to course 
objectives. 


appropriate for their reading 
level, and read and learn 

content from those materials 
related to course objectives. 


1-2 days per week in at least 
50% of class periods, teacher 
helps students select MCLA 
materials appropriate for their 
reading level, and read and 
learn content from those 
materials related to course 
objectives. 


Occasionally (at least monthly 
in at least 1 class period), 
teacher helps students select 
MCLA materials appropriate 
for their reading level, and read 
and learn content from those 
materials related to course 
objectives. 


Teachers do not help 
students select MCLA 
materials appropriate for 
their reading level, and 
read and learn content 
from those materials 
related to course 
objectives. 
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Memphis City Schools / University of Memphis 


Memphis, Tennessee 


B.4. Using different instructional materials 


a 


b 


d 


e 


B.4.c. Use of materials that the 
teacher has collected from 
other sources (e.g., on- 
line/libraries, etc.): At least 3 
days per week (and in all class 
periods, teacher collects 
materials from other sources 
and help students to read and 
learn content from those 
sources related to course 
objectives. 


1-2 days per week in at least 
75% of class periods, 
teacher collects materials 
from other sources and help 
students to read and learn 
content from those sources 
related to course objectives. 


1-2 days per week in at least 
50% of class periods, teacher 
collects materials from other 
sources and helps students to 
read and learn content from 
those sources related to course 
objectives. 


Occasionally (at least monthly 
in at least 1 class period), 
teacher collects materials from 
other sources and help students 
to read and learn content from 
those sources related to course 
objectives. 


Teachers do not collect 
materials from other 
sources and help students 
to read and learn content 
from those sources related 
to course objectives. 


B.5. Using cooperative learnin 


activities with students 


a 


b 


c 


d 


e 


B.5.a. Frequency of 
cooperative learning activities 


in class periods: Teacher 
includes cooperative learning 
activities as part of lessons at 
least 3 days per week in all 
class periods. 


Teacher includes 
cooperative learning 
activities as part of lessons 
1-2 days per week in at least 
75% of class periods. 


Teacher includes cooperative 
learning activities as part of 
lessons 1-2 days per week in at 
least 50% of teacher’s class 
periods. 


Teacher includes cooperative 
learning activities as part of 
lessons occasionally (at least 
monthly) in at least 1 class 
period. 


Teacher uses whole group 
instruction with no 
evidence of cooperative 
learning activities. 
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Innovation Configuration Map for the Memphis Content Literacy Academy (MCLA) Program 
Memphis City Schools / University of Memphis 
Memphis, Tennessee 


B.5. Using cooperative learning activities with students 


a b c d e 
B.5.b. Purposes of cooperative | Teacher uses cooperative Teacher uses cooperative Teacher uses cooperative There is no evidence that 
learning activities: Teacher learning activities to provide | learning activities to provide learning activities to provide teacher uses cooperative 
uses cooperative learning students opportunities to students opportunities to students with practice of new learning activities or 
activities to provide students practice the specific literacy | practice the specific literacy content knowledge and skills. cooperative learning 
opportunities to practice strategies with various texts | strategies only with their activities have no clear 
extensively both their use of and separately to practice adopted textbook and to learning objectives. 
specific literacy strategies with | their application of new practice their application of 
various texts and their content knowledge and new content knowledge and 
application of new content skills (i.e., to “over-learn” skills. 
knowledge and skills those strategies, that 
(i.e., to “over-learn” those knowledge, and those 
strategies, that knowledge, and | skills). 
those skills). Cooperative 
learning strategies also are 
used to differentiate instruction 
based on identified learning 
needs. 


B.6. Collaborative Teacher Work ((schedule facilitates collaborative work, core content teachers regularly develop collaborative instructional plans) 


a b c d e 
B.6.a. Breadth of teacher At least 75% of MCLA 50% - 74% of MCLA 30% - 49% of MCLA Fewer than 30% of MCLA 
participation in collaborative | teachers in a department teachers in a department teachers in a department teachers in a department 
planning: All MCLA teachers | and/or grade level team have | and/or grade level team have | and/or grade level team have | and/or grade level team have 
in a department and/or grade | time each week to work time each week to work time each week to work time each week to work 
level team have time each collaboratively on integrating | collaboratively on integrating | collaboratively on integrating | collaboratively for on 
week to work collaboratively | literacy strategies into their literacy strategies into their literacy strategies into their integrating literacy strategies 
(with each other and literacy content lesson plans. content lesson plans. content lesson plans. into their content lesson 
coaches?) on integrating plans. 
literacy strategies into their 
content lesson plans. 
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Memphis, Tennessee 


B.6. Collaborative Teacher Work ((schedule facilitates collaborative work, core content teachers regularly develop collaborative instructional plans) 


a 


b 


c 


d 


e 


B.6.b. Frequency/duration of 
collaborative planning: 
MCLA teachers in a 
department and/or grade level 
team work collaboratively for 
useful periods of time 
(minimum 45 minutes 
weekly) on integrating 
literacy strategies into their 
content lesson plans. 


MCLA teachers in a 
department and/or grade level 
team work collaboratively for 
useful periods of time 
(minimum 30 minutes at least 
twice monthly) on integrating 
literacy strategies into their 
content lesson plans. 


MCLA teachers in a 
department and/or grade level 
team work collaboratively for 
useful periods of time 
(minimum 30 minutes at least 
once per month) on 
integrating literacy strategies 
into their content lesson 
plans. 


MCLA teachers in a 
department and/or grade level 
team work collaboratively 
less than once per month for a 
minimum of 30 minutes on 
integrating literacy strategies 
into their content lesson 
plans. 


MCLA teachers in a 
department and/or grade level 
team work collaboratively for 
less than 15 minutes and no 
more than twice monthly on 
integrating literacy strategies 
into their content lesson 
plans. 
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Innovation Configuration Map for the Memphis Content Literacy Academy (MCLA) Program 
Memphis City Schools / University of Memphis 


C. SCHOOLWIDE FACTORS 


Memphis, Tennessee 


C.1. Principal Leadership 


a 


b 


c 


d 


e 


C.1.a. Attending MCLA 
events: Principal and assistant 
principals attend both types 
of MCLA-related events for 
teachers: kick-off and 
laureate conference. 


Principal attends both types 
of MCLA-related events for 
teachers and a team member 
(e.g., assistant principal) 
attends one event. 


Principal attends all MCLA- 
related events for teachers 
without any team members. 


Principal sends a designee to 
attend one or both types of 
MCLA-related events for 
teachers. 


No school administrator 
attends MCLA-related events 
for teachers. 


C.1.b. Communicating within 
the school the importance of 
literacy instruction in content 
areas: At weekly faculty 
meetings and at least once 
weekly during daily 
announcements, principal 
communicates to the teachers 
and students his or her belief 
in the importance of literacy 
instruction for improving 
student achievement in the 
content areas. 


At least twice monthly at 
faculty meetings and during 
daily announcements, 
principal communicates to the 
teachers and students his or 
her belief in the importance 
of literacy instruction for 
improving student 
achievement in the content 
areas. 


At least once monthly at 
faculty meetings and during 
daily announcements, 
principal communicates to the 
teachers and students his or 
her belief in the importance 
of literacy instruction for 
improving student 
achievement in the content 
areas. 


Every other month, principal 
communicates to the teachers 
his or her belief in the 
importance of literacy 
instruction for improving 
student achievement in the 
content areas. 


Principal communicates to 
the teachers his or her belief 
in an alternative view of what 
kinds of instruction is 
important for improving 
student achievement in the 
content areas (i.e., works as a 
saboteur of MCLA) 
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Memphis City Schools / University of Memphis 


Memphis, Tennessee 


C.1. Principal Leadership 


a 


b 


c 


d 


e 


C.1.c. Communicating to 
broader school community 
the importance of literacy 
instruction in content areas: 
Principal has and implements 
an integrated, multifaceted 
approach to continually 
communicate to parents and 
other stakeholders via 
multiple avenues (e.g., 
banners, posters, newsletters, 
speaking at events with 
parents/ community members 
present) his or her belief in 
the importance of literacy 
instruction for improving 
student achievement in the 
content areas. 


Principal has a plan to 
communicate with parents 
and other stakeholders his or 
her belief in the importance 
of literacy instruction for 
improving student 
achievement in the content 
areas but only partially 
implements that plan. 


Without a communication 
plan, principal sometimes 
communicates to parents and 
other stakeholders his or her 
belief in the importance of 
literacy instruction for 
improving student 
achievement in the content 
areas. 


Principal communicates to 
parents and other 
stakeholders his or her belief 
in the importance of 
improving student 
achievement in the content 
areas without reference to 
literacy instruction. 


Principal communicates to 
parents and other 
stakeholders his or her belief 
in an alternative view of what 
kinds of instruction is 
important for improving 
student achievement in the 
content areas (i.e., works as a 
saboteur of MCLA). 


C.1.d. Participation in MCLA 
Fellowship: Principal and 
other administrators 
participate actively in all 
MCLA Fellowship meetings. 


Principal attends all MCLA 
Fellowship meetings and 
brings a team member to 
most of the meetings. 


Principal participates actively 
in all MCLA Fellowship 
meetings but does not bring a 
team member. 


Principal attends almost all 
Fellowship meetings and 
ensures that team members 
attend missed meeting(s). 


The principal participates 
sporadically in MCLA 
Fellowship meetings. The 
school is not represented at 
every meeting. 


C.1.e. Incorporation of 
literacy and MCLA in 


improvement plan: Principal 
ensures that schoolwide 
literacy instruction in content 
area classes and the MCLA 
project are a priority in the 
school’s improvement plan. 


Principal ensures that 
schoolwide literacy 
instruction in content area 
classes and the MCLA 
project are included in the 
school’s improvement plan. 


Principal ensures that 
schoolwide literacy 
instruction in content area 
classes is included in the 
school’s improvement plan 
without any mention of 
MCLA. 


The school improvement plan 
emphasizes content-area 
instruction without a focus on 
literacy. 
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Memphis, Tennessee 


C.1. Principal Leadership 


a 


b 


c 


d 


C.1.f. Reallocation and 
procurement of additional 
resources: Principal 
reallocates existing resources 
and seeks additional 
resources to supplement and 
support schoolwide MCLA 
implementation. 


Principal reallocates existing 
resources but does not seek 
additional funding for MCLA 
and schoolwide literacy 
efforts. 


Principal expects existing 
resources to cover the costs of 
resources to support MCLA 
implementation. 


Principal reallocates MCLA 
resources to other purposes 
not related to literacy 
instruction in the content 
areas. 


C.1.g. Supportive schedule: 
The school schedule enables 
all grade-level teachers in a 
department or grade level 
teams to work collaboratively 
for useful periods of time 
(minimum of 45 minutes 
weekly) during the regular 
school day. 


The school schedule enables 
at least 75% of grade-level 
teachers in a department or 
grade level team to work 
collaboratively for useful 
periods of time on integrating 
literacy strategies into their 
content lesson plans during 
the regular school day.. 


The school schedule enables 
50% - 75% of grade-level 
teachers in a department or 
grade level team to work 
collaboratively for useful 
periods of time on integrating 
literacy strategies into their 
content lesson plans during 
the regular school day.. 


The school schedule enables 
30% - 49% of grade-level 
teachers in a department or 
grade level team to work 
collaboratively for useful 
periods of time on integrating 
literacy strategies into their 
content lesson plans during 
the regular school day.. 


The school schedule does not 
enable grade-level teachers in 
a department or grade level 
team to work collaboratively 
for useful periods of time on 
integrating literacy strategies 
into their content lesson plans 
during the regular school 
day.. 


C.2. Administrator Walkthroughs 


a 


b 


c 


d 


e 


C.2.a. Frequency of 
walkthroughs: Administrator 
does daily walkthroughs of 
core content classes. 


Administrator does at least 
2x/weekly walkthroughs of 
core content classes. 


Administrator does 1x weekly 
walkthroughs of core content 
classes. 


Administrator does at least 
monthly (but < weekly) 
walkthroughs of core content 
classes. 


Administrator never does 
walkthroughs of core content 
classes. 


C.2.b. Purpose of 
walkthroughs: When the 
administrator performs 
informal walkthroughs, he or 
she looks for student use of 
literacy strategies. 


When the administrator 
performs informal 
walkthroughs, he or she looks 
for teacher use of literacy 
strategies. 


When the administrator 
performs informal 
walkthroughs, he or she looks 
for general, nonspecific, 
superficial use of literacy 
strategies. 


When the administrator 
performs informal 
walkthroughs, he or she is 
focused on other aspects of 
teacher performance and not 
use of literacy strategies. 


Administrator walkthroughs 
only happen for formal 
evaluations. 
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Memphis, Tennessee 


C.2. Administrator Walkthroughs 


a 


b 


c 


d 


e 


C.2.c. Feedback provided 
from walkthroughs: When the 
administrator observes 
student use of a literacy 
strategy during a 
walkthrough, the 
administrator provides 
immediate feedback to the 
teacher on teacher and 
student use of the strategy. 


When the administrator 
performs walkthroughs, he or 
she provides feedback (not 
immediate but fairly soon 
after) to the teacher on 
student and teacher use of 
literacy strategies. 


When the administrator 
performs walkthroughs, he or 
she provides feedback (not 
immediate but fairly soon 
after) to the teacher on 
teacher use of literacy 
strategies 


When the administrator 
observes the use of a literacy 
strategy during a 
walkthrough, the 
administrator acknowledges 
that use to the teacher. 


Even when the administrator 
observes the use of a literacy 
strategy during a 
walkthrough, the 
administrator does not 
acknowledge that use to the 
teacher. 


C.3. Principal’s Support of Coach (inclusion in leadership team meetings, classroom implementation of MCLA strategies, influencing 


allocation of resources related to literacy; coaching role) 


a 


b 


c 


d 


e 


C.3.a. Principal includes 
coach in leadership meetings: 
Principal regularly includes 
coach in leadership team 
meetings. 


Principal sometimes includes 
coach in leadership team 
meetings. 


Principal infrequently 
includes coach in leadership 
team meetings. 


Principal does not include 
coach in leadership team 
meetings. 


Principal prevents coach from 
attending leadership team 
meetings. 


C.3.b. Principal 
communicates expectations to 


teachers regarding working 
with coach: Principal actively 
and consistently 
communicates to teachers the 
expectation that they work 
with their coach to support 
classroom implementation of 
MCLA strategies. 


Principal sometimes 
communicates to teachers the 
expectation that they work 
with their coach to support 
classroom implementation of 
MCLA strategies. 


Principal does not explicitly 
communicate to teachers the 
expectation that they work 
with their coach to support 
classroom implementation of 
MCLA strategies but allows 
it. 


Principal communicates 
expectations that teachers’ 
efforts be directed towards 
alternatives to classroom 
implementation of MCLA 
strategies (related or 
unrelated to literacy). 
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Memphis, Tennessee 


C.3. Principal’s Support of Coach (inclusion in leadership team meetings, classroom implementation of MCLA strategies, influencing 


allocation of resources related to literacy; coaching role) 


a 


b 


c 


d 


C.3.c. Principal views coach 
as resource for literacy 
related decisions: Principal 
views the coach as a resource 
and regularly seeks coach 
input on decisions related to 
literacy: curricula and 
instruction, material 
purchases, and assessments. 


Principal views the coach as a 
resource and selectively seeks 
coach input on decisions 
related to literacy: curricula 
and instruction, material 
purchases, and assessments. 


Principal does not view the 
coach as a resource and does 
not seek the coach’s advice in 
making decisions related to 
literacy. 


C.3.d. Principal views coach 
as resource for school PD: 
Principal requests that the 
coach facilitate school-based 
staff PD in use of MCLA 
strategies. 


Principal requests that the 
coach provide information to 
staff (but is not asked to 
provide PD) about MCLA 
strategies. 


Principal requests that the 
coach provide information to 
him/her (but not school staff) 
about MCLA strategies. 


Principal requests that the 
coach perform duties outside 
their defined role (e.g., 
substitute teaching, cafeteria 
duty, etc.). 


Principal requires that the 
coach spend most of their 
time on duties outside their 
defined role (e.g., substitute 
teaching, cafeteria duty, etc.). 


C.4. School Culture (core content teachers’ acceptance of collective responsibility for student literacy, core content area teachers describe 
literacy instruction within the content areas as a school priority, use by core content teachers of a widely accepted research-based vocabulary 
related to literacy instruction/literacy strategies) 


a 


b 


c 


d 


e 


C.4.a. Collective 
responsibility for student 
literacy. At least 90% of 
content area teachers can 
describe how they are 
working with grade-level 
content area colleagues to 
integrate literacy instruction 
into their content lesson 
plans. 


75% - 89% of content area 
teachers can describe how 
they are working with grade- 
level content area colleagues 
to integrate literacy 
instruction into their content 
lesson plans. 


50% - 74% of content area 
teachers can describe how 
they are working with grade- 
level content area colleagues 
to integrate literacy 
instruction into their content 
lesson plans. 


25% - 49% of content area 
teachers can describe how 
they are working with grade- 
level content area colleagues 
to integrate literacy 
instruction into their content 
lesson plans. 


Less than 25% of content area 
teachers can describe how they 
are working with grade-level 
content area colleagues to 
integrate literacy instruction 
into their content lesson plans. 
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C.4. School Culture (core content teachers’ acceptance of collective responsibility for student literacy, core content area teachers describe 
literacy instruction within the content areas as a school priority, use by core content teachers of a widely accepted research-based vocabulary 
related to literacy instruction/literacy strategies) 


a 


b 


c 


d 


e 


C.4.b. Schoolwide use of 
literacy strategies. At least 
90% of core content area 
teachers incorporate student 
use of literacy strategies as 
an integral part of content 
area instruction. 


75% - 89% of core content 
area teachers incorporate 
student use of literacy 
strategies as an integral part 
of content area instruction. 


50% - 74% of core content 
area teachers incorporate 
student use of literacy 
strategies as an integral part 
of content area instruction. 


25% - 49%of core content 
area teachers incorporate 
student use of literacy 
strategies as an integral part 
of content area instruction. 


Less than 25% of core content 
area teachers incorporate 
student use of literacy 
strategies as an integral part of 
content area instruction. 


C.5. Critical mass of core content teachers (significant proportion of core content teachers participating in MCLA, percentage of students that have MCLA 
trained teachers for all four of their core content areas) 


a 


b 


c 


d 


e 


C.5.a. Proportion of core 
content teachers in MCLA: 
At least 90% of the core 
content teachers in the school 
are participating in the 
MCLA project. 


Between 67% and 89% of the 
core content teachers in the 
school are participating in the 
MCLA project. 


50% to 66% of the core 
content teachers in the school 
are participating in the 
MCLA project. 


Between 33 and 49% of the 
core content teachers in the 
school are participating in the 
MCLA project. 


Less than 33% of the core 
content teachers in the school 
are participating in the 
MCLA project. 


C.5.b. Percentage of students 
having MCLA-trained 
teachers: Between 80 and 
100% of the students have 
MCLA trained teachers for 
all four of their core content 
areas. 


Between 60 and 79% of the 
students have MCLA trained 
teachers for all four of their 
core content areas. 


Between 40 and 59% of the 
students have MCLA trained 
teachers for all four of their 
core content areas. 


Between 20 and 39% of the 
students have MCLA trained 
teachers for all four of their 
core content areas. 


Between 0 and 19% of the 
students have MCLA trained 
teachers for all four of their 
core content areas. 
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Summary Sheet for the MCLA Innovation Configuration Map 


| A.2. Student Use of Literacy Strategies 


pe pepe [o>] [wom 
le 
re 
ae ng TE POE aE TEC 


B.1. Core Content Teachers Use Information from Assessments for Revising Instructional Plans and Developing 
| PlansforIntervention UU 


pe pe De] ae 


B.1.d.Design interventions for struggling students 


re 
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B.2. Core Content Teachers Provide Explicit, Direct Instruction and Practice 


B.2.a. Introduction of strategies 


ee ee ee ee 
B.2.c. Guided practice 
Se 


B.2.d. Independent use 


A 


B.2.e. Differentiated instruction 


A 


B.3. Objectives of Instructional Plans 


B.3.a. Objectives in terms of core content standards’ learning objectives, knowledge, and skills 


Ps fe | « | © | + [some 


B.3.b. Objectives in terms of literacy strategies 


ee 


B.4. Using Different Instructional Materials 


B.4.a. Use of adopted textbook 


ee 


B.4.b. Use of MCLA supplementary materials 
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Innovation Configuration Map for the Memphis Content Literacy Academy (MCLA) 


Program 
Memphis City Schools / University of Memphis 
Memphis, Tennessee 


a 


B.4.c. Use of materials that the teacher has collected from other sources 


ee 


B.5 Using Cooperative Learning Activities with Students 


B.5.a. Frequency of cooperative learning activities in class periods 


Pos [Te [< | >] * [emer 


B.5.b. Purposes of cooperative learning activities 


a 


B.6. Collaborative Teacher Work 


B.6.a. Breadth of teacher participation in collaborative planning 


ee 


B.6.b. Frequency/duration of collaborative planning 


ee 


Property of Memphis City Schools / University of Memphis Bond Building Room 232, 2930 Airways Blvd, Memphis, TN 38116 
[901.416.2931]. Contact Elizabeth Heeren, Ed.D. (heerenelizabeth@mesk12.net) for the latest version. 

NOT TO BE REPRINTED WITHOUT PERMISSION 
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Innovation Configuration Map for the Memphis Content Literacy Academy (MCLA) 


Program 
Memphis City Schools / University of Memphis 
Memphis, Tennessee 


C.1. Principal Leadership 


C.1.a. Attending MCLA events 


ee 


C.1.b. Communicating within the school the importance of literacy instruction in content areas 


ee 


C.1.c. Communicating to broader school community the importance of literacy instruction in content areas 


ee 


C.1.d. Participation in MCLA Fellowship. 


C.1.e. Incorporation of literacy and MCLA in improvement plan 


a 


C.2. Administrator Walkthroughs 


C.2.a. Frequency of walkthroughs 


ee 


C.2.b. Purpose of walkthroughs 


ee 


C.2.c. Feedback provided from walkthroughs 


Property of Memphis City Schools / University of Memphis Bond Building Room 232, 2930 Airways Blvd, Memphis, TN 38116 
[901.416.2931]. Contact Elizabeth Heeren, Ed.D. (heerenelizabeth@mesk12.net) for the latest version. 
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Innovation Configuration Map for the Memphis Content Literacy Academy (MCLA) 


Program 
Memphis City Schools / University of Memphis 
Memphis, Tennessee 


fr 


C.3. Principal’s Support of Coach 


C.3.a. Principal includes coach in leadership meetings 


ee 


C.4. School Culture 


C.4.a. Collective responsibility for student literacy 


ee 


C.4.b. Schoolwide use of literacy strategies 


ee 


C.5. Critical Mass of Core Content Teachers 


C.5.a. Critical mass of core content teachers 


ee 


C.5.b. Critical mass of MCLA-trained teachers for students 


ee 


Property of Memphis City Schools / University of Memphis Bond Building Room 232, 2930 Airways Blvd, Memphis, TN 38116 
[901.416.2931]. Contact Elizabeth Heeren, Ed.D. (heerenelizabeth@mesk12.net) for the latest version. 

NOT TO BE REPRINTED WITHOUT PERMISSION 
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Appendix B-1. MSRP Evaluation Report, Y4 March 1, 2011. Page 1 of 5 
READ 180 Classroom Observation Protocol 


Observer: Observation date: 

School: Teacher: Grade a6 ag Sale 
Official class start time: Official class end time: 

Observation start time: Observation end time: 

Number of students 15 minutes after class start time: Number of boys: | 

Number of non-black students: Number of girls: 


Whole-group instruction: 


1. Did the teacher provide whole-group instruction? yes no 


Did the teacher use identifiable READ 180 materials or lessons? Clyes Ono 


low mixed O high 


2 
3. What was the level of engagement? 
4 


Did the teacher use any of the following activities? (check all that apply, then in O shared reading 
the space below, list any specific reading strategies that you recognize.) 0 think-alouds 
Oo a I ea eet arate to abet teas ete tee. ee eae ae modeling of reading 
strategies 


5. What were the start and end times of READ 180 whole-group instruction? 


6. Briefly describe what occurred during whole-group instruction. 


Overall student rotations (answer these questions by observing all small groups): 


7. Did students separate into small groups? Oyes Ono 
: Group A: 
8. How many students were in each group? (If there were only two Gun! 
groups, please draw a line through “Group C.”’) Pe 
GroupC: > 
computer use yes no 
10. Did the teacher and students use READ 180 materials or lessons? —_ independent reading yes no 
small-group instruction yes no 
Rotation 1: 
11. What were the start and end times of each rotation? Rotation 2: 
Rotation 3: 


12. What were the levels of engagement? (Ifa small group did not participate in one or more rotations, please draw a 
line through the corresponding “low — mixed — high” space/s.) 


Group A Group B Group C 
computer use low Omixed OC high low Omixed OC high low Omixed OC high 
independent reading low Omixed DO high low Omixed ODO high low Omixed DO high 
small-group instruction low Omixed DO high low Omixed O high low Omixed OD high | 


Engagement Ranges: Low: <30% of students working | Mixed: 30-80% of students working | High: >80% of students working 
R&S Coffey, D., & Feldman, J. (2009). READ 180 Observation Protocol. Philadelphia, PA: Research for Better Schools. Page 1 of 5 
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Small-group rotations (answer these questions according to the one group you followed in detail): 


Small-group instruction rotation 


13. Did the teacher provide small-group instruction? (whether or not students were engaged) Oyes Ono 


13a. Did the teacher and/or = ieee ere 13b. Did the teacher and/or O fluency 
lesson specifically address — ee ti lesson specifically address 0 vocabulary 
any of the following 7 he sees bis oF Fe any of the following 1 comprehension 
reading strategies? = ae ont penetaled ucelons reading domains? CX phonics 
other 
14. Were the students expected to use rBooks? Oyes Ono 
15. Were the students expected to use other books or worksheets? Oyes (Ono 


15a. If yes, please describe what other materials the students used (provide as much information as possible, 
e.g., author, title, series, publisher—be sure to look for Scholastic logo or other indicators) 


Independent reading rotation 


17. Were students expected to read and/or listen to Scholastic’s READ 180 books and/or _ 


audiobooks? = ee 

17a. Are Scholastic novels and trade books easily accessible to students? 1 yes no 

17b. Are books organized according to reading (i.e., SRI or lexile) level? yes Ono 
18. Did the students read other independent reading materials? Clyes Ono 


19. If the students read other materials, please describe those, including title and author whenever possible. 


Engagement Ranges: Low: <30% of students working | Mixed: 30-80% of students working | High: >80% of students working 
R&S Coffey, D., & Feldman, J. (2009). READ 180 Observation Protocol. Philadelphia, PA: Research for Better Schools. Page 2 of 5 
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Computer workstations rotation 


March 1, 2011. Page 3 of 5 


21. Did all students present have access to a working computer running READ 180 


software for the majority of their computer rotation? = - 
22. Did students seem to understand how to complete the READ 180 lessons on the _ _ 
computer? a 
23. Briefly describe what occurred during the computer workstations rotation. 
Whole-group wrap-up: 
24. Did the teacher use identifiable READ /80 materials or lessons? (Look for the Scholastic _ yes Ono 


logo or evidence of use of any from the Teacher Bookshelf, not just rBooks.) 


25. What was the level of engagement? 


Olow Omixed Ohigh 


26. What were the start and end times of whole-group wrap-up? 


27. Which of the following features did the teacher lead or expect during 


wrap-up? (check all that apply) 


0 clean-up 


0 “exit ticket” related to classwork 
1 review of class or lesson 
1 connection of lesson with another 


content area or previous classes 


28. Briefly describe what occurred during whole-group wrap-up. 


Questions to ask the teacher immediately after the observation: 


29. How many students were absent today? 


30. Do your students typically use rBooks? Oyes Ono 

If the answer to #30 was “yes,” ask 31: 

31. What rBook Workshop number is this class working on? a 

31a. About how frequently do you use more than weekly twice monthly 1 quarterly 
rSkills tests and quizzes? weekly monthly 1 never 

31b. About how frequently do you use SAM, more than weekly twice monthly 1 quarterly 
the Scholastic Achievement Manager? weekly monthly 1 never 


Engagement Ranges: Low: <30% of students working | Mixed: 30-80% of students working | High: >80% of students working 
R&S Coffey, D., & Feldman, J. (2009). READ 180 Observation Protocol. Philadelphia, PA: Research for Better Schools. Page 3 of 5 
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Questions to ask the teacher immediately after the observation, contd.: 


31c. Which SAM reports do you use? 


If personal audio equipment (CD or cassette players, headphones, etc.) was available, but no students used it, ask the following: 


32. Do the CD players and headphones for “books-on-CD” guided reading work properly? all some none 


33. Is there anything in particular I should know about this class session or these students? 


33a. What did today’s lesson tell you about what your students are learning and still need to learn? What will students 
in this class being doing over the next few weeks? 


33b. What challenges have you faced in encouraging your students to be actively engaged in READ 180? How have 
you approached these challenges? 


Engagement Ranges: Low: <30% of students working | Mixed: 30-80% of students working | High: >80% of students working 
R&S Coffey, D., & Feldman, J. (2009). READ 180 Observation Protocol. Philadelphia, PA: Research for Better Schools. Page 4 of 5 
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Questions for the observer to answer immediately after the observation: 


C1 less than 60 minutes 
1 60 to 80 minutes 
81 to 87 minutes 
88 to 95 minutes 
C1 96 or more minutes 


34. What was the length of time students were expected to be working during 
the observed READ 180 section (i.e., length of class time minus all of the 
following that occurred: time for arrival, getting ready or lining up for 
dismissal, or other events that took up class time)? 


35. Did the classroom have designated spaces for whole-group and small-group 
instruction, independent reading, and computer workstations (i.e., spaces that yes no 
were made distinct by signs or furniture arrangement)? 


36. Did at least five of the computers appear to work and run READ 180 software? Oyes Ono 

37. Did students seem to have access to personal audio equipment (e.g., tape 1 yes, 4 to 7 individual players 
players, CD players, or additional computers; headphones) during 1 yes, | to 3 individual players 
independent reading? (Please ask the teacher if no students used audio C1 yes, 1 player; many headphones 
equipment.) 1 no 

38. Did students seem to have a clear idea of what was expected of them 
during rotations (e.g., students went to rotation “centers” as if computer use yes no 
accustomed; students easily identified what books they were reading or independent reading yes no 
what computer lessons they were on; students expected guidance from small-group instruction 0 yes no 


the teacher during small-group instruction)? 


Additional Notes: 


Engagement Ranges: Low: <30% of students working | Mixed: 30-80% of students working | High: >80% of students working 
R&S Coffey, D., & Feldman, J. (2009). READ 180 Observation Protocol. Philadelphia, PA: Research for Better Schools. Page 5 of 5 
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OBSERVATION CHECKLIST 
MEMPHIS CITY SCHOOLS 


Observer: Teacher: 
Date: School: 
Objective: 


Note: Information with an asterisk (*) may be obtained through teacher interview or review of 
student reports. 


1 2 3 4 

INITIAL IMPRESSION Not at All Somewhat Mostly Completely 

1. Teacher has been trained and is effectively 
managing and interacting with students 
according to the model. * 

2. READ 180 books, trade books, and classroom 
libraries are complete, leveled and accessible. 

3. Students are actively engaged in whole group 
instruction or a rotation. 

4. At least five computer workstations are functional 
and READ 180 is operational for students. 

5. Teacher workstation is functional and SMS is 
installed and operational. 

6. Room is arranged with appropriate space for 
independent reading, small group instruction, 
whole group instruction, and computer 
workstations. 


ORGANIZATION YES NO 


There are 21 or less in the room. 


2. Small group rotations consist of seven students 
or less. 


1 2 3 4 
Not at All Somewhat Mostly Completely 


3. Furniture is arranged to allow for effective 
rotations. 

4. Students have been placed according to initial 
SRI and are grouped flexibly according to data 
generated by reports. * 

5. Classroom management is effective and rotations 
move smoothly. 

6. Reports generated by the READ 180 software are 


available for progress monitoring. * 

7. Student reading logs, portfolios, journals, etc. 
are organized and accessible. 

8. The total class period is arranged according to 
the READ 180 model: 20 minutes whole group 
instruction, three 20-minute rotations (small 
group instruction, independent reading, 
individual software instruction), and wrap-up 
activities. 
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1 2 3 4 


WHOLE Group INSTRUCTION Not at All Somewhat Mostly | Completely 


1. Teacher provides whole group direct instruction 
using modeling, shared reading, or mini-lessons. 


2. Teacher uses READ 180 resources for instruction. * 


3. Teacher brings closure to the class based on the 
day’s objective; reviewing, reading aloud, building 
community, sharing, discussing, or reflecting. 

4. Students use Rbooks or Flexbooks for skill 
development. 


SMALL GROUP ROTATION 


1. Teacher interacts with students in small groups; 
focusing on skills, comprehension, and discussion 
related to the day’s objective. 

2. Teacher differentiates instruction based on data 
generated by READ 180 software reports. * 

3. Students are actively involved in mini-lessons, 
discussions, writing activities, reflections, skill 
practice, etc., using READ 180 resources. 


COMPUTER ROTATION 


1. Students work on READ 180 software for twenty 
minutes. * 


2. Students are actively engaged in learning and 
remaining on task. 


INDEPENDENT READING ROTATION 


1. Students are actively engaged in Independent 
reading. 

2. Students successfully pass Reading Counts quizzes 
indicating comprehension of independent reading 
materials. * 

3. Teacher monitors student logs, reflections, RC 
quizzes, quick writes, and reports to ensure 
effective independent reading behavior. * 


Signature of observer: 


OBSERVATION NOTES 
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Appendix C: Equations and Notes for READ 180 Observation Ratings 


Data was gathered from RBS and MCS protocols on the 15 topics listed in the body of the report. 
Subsequently, information from the different components was combined as follows. 


Component Weight 

SG: Small-group rotation ...........ccecceeescceeseeeeceeeseeeeeeeeeeeeeeeeeeeeesaeeeseeenseeens (2 items) 4 units 
CR: Computer rotation .........ccceecccecescessccceeseeeseeceeceeseaeeceaeeeeeeceseeesnneesnaeensaes (3 items) 4 units 
WGI: Whole-group instruction, including Wrap-Up ...........ccceeeeesseeeereeees (4 items) 3 units 
Base (including classroom space and layout, number of students 

enrolled, timing, and class atmosphere)............:c:cccessceeseeeeeeeeeteeeeneeeeseeeeaes (4 items) 2 units 
IR: Independent reading rotation ...........ceeeceeescceeeeeeeeeeeeeeeeneeeeneeeseeeeesaeenes (1 item) 1 unit 


To calculate the weighted component ratings, the following equations were used 

Base = (total of scores on 4 items) / 2 

WGI = (total of scores on 4 items) * .75 

SG = (total of scores on 2 items) * 2 

CR = (total of scores on 3 items ) / .75 

IR = (score on one item) 

OR = overall rating 

The overall observation rating was calculated by totaling the weighted scores for the above five 
components of READ 180. The total possible points was 56, which was divided by 14 (total number of 


items used). This resulted in a number between 0 and 4, which was used as the classroom observation 
score. 


(Base + WGI+SG+CR+IR) 
14 


OR= 
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Appendix D 
Specification of the Multi-Level (Cross-Sectional) Regression Models Employed 
to Test the One-Year and Two-Year Impacts of the READ 180 Intervention in 
and Differences in READ 180 Impact in MCLA Treatment and Control Schools 


READ 180 Impact 


Three multi-level regression models were employed to conduct cross-sectional analyses of the 
immediate and long-term, two-year impact of READ 180 on student reading and subject area 
achievement at the end of Years 1 and 2 of the Memphis Striving Readers study. The first model 
estimated the immediate impact of READ 180 for students in grades 6-8 in Year 1. 


At the student level, 


M 
Yi = Boj + PiXiy t+ B2X 2 + P3X3i + PsX ay + S Bons X (m+4)i + Ti, 
m=1| 
where 
Y; is the Spring Year 1 test score (ITBS/TCAP) for student 7 in school /; 


Xj, 18 an uncentered dummy variable coded 0 for READ 180 control and 1 for READ 180 
treatment students in school /; 


Xj 18 a grand mean centered baseline test score (ITBS/TCAP) for student 7 in school /; 


3, 18 a grand mean centered dummy variable coded | for students in 7" grade and 0 
otherwise; 


X4j 18 an grand mean centered dummy variable coded 1 for students in 8"" grade and 0 
otherwise; 


Xm+4)ij 18 the m'" of M additional student-level covariates that may be included in the final 
model depending on whether they satisfy criteria for inclusion; 


fo; is the adjusted mean Spring test score for the control students in school /, controlling 
for the baseline 2006 test score and other covariates; 


f, is the adjusted difference between READ /80 treatment and control group mean Spring 
test scores (the READ /80 treatment effect), controlling for the baseline test score and 
other covariates; 


f> is the slope of the regression of Spring test scores on baseline test scores; 
Bs is the adjusted difference between the mean 6" and 7" grade Spring test scores; 


fa is the adjusted difference between the mean 6" and 8" grade Spring test scores; 


‘m+4 18 the coefficient for the m" of M additional student-level covariates that may be 
y 


included in the final model depending on whether they satisfy criteria for inclusion; 
and 


rj is a unique effect for student i in school 7 and is ~ N(0,0°); 
All of the above coefficients at the student level, except fo;, are assumed constant across schools. 


fo;, the adjusted mean Spring Year | test score for the control students in school 7, is modeled as 
a function of school-level covariates: 
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P 
Boj= yoo+ S yopWri + u0j, 
p=! 


where 


W,,; is the p of P school-level covariates that may be included in the final model 
depending on whether they satisfy criteria for inclusion; 


yoo is the adjusted mean Spring test score for all control students; 


Yop 1s the coefficient for the p" of P school-level covariates that may be included in the 
final model depending on whether they satisfy criteria for inclusion; and 


Uo; is the unique effect of school 7 and is ~N(0,1). 


The null hypothesis of no READ 180 treatment effect on Spring test scores is Ho: (1 = 0 and is 
tested with a f-statistic. 


The second model estimated the immediate impact for students in each grade separately in 
Year | and for sixth-grade students in Year 2. The only difference in this second model was the 
exclusion of the two dichotomous covariates designating whether or not students were enrolled 
in grades 7 or 8. These covariates were not needed in the model used for students in a single 
grade. 


The third model estimated the long-term, two-year impact of READ /80 on students in grades 6 
and 7 in Year | and in grades 7 and 8 in Year 2 (referred to as “stayers” in the report). The only 
difference in this third model from the first was the inclusion of only one dichotomous covariate 
designating whether or not students were enrolled in grade 8 in Year 2. The second model was 
also used to estimate the long-term, two-year impact of READ 180 separately for the students 
going from sixth grade to seventh grade and for those going from seventh grade to eighth grade. 


Differences in READ 180 Impact in MCLA Treatment and Control Schools 


The above three models were modified slightly to estimate the difference between the impacts of 
READ 180 in MCLA treatment and control schools. The modification was to include a 
dichotomous, school-level variable indicating whether the school was an MCLA treatment or 
control school. At the school level, this variable was included in the specification of the adjusted 
mean of the READ 180 control students and of the estimate of the READ 180 impact. 


At the student level, the coefficient estimating the READ 180 was assumed to vary across 
schools: 
M 
Yy = Boj + BiXiy + B2X aw + BsXay + BaX ay + S Bons X (medi + Vi. 


m=| 


where 


fi; is the adjusted difference between READ 180 treatment and control group mean 
Spring test scores (the READ /80 treatment effect) for school j, controlling for the 
baseline test score and other covariates. 


This allows modeling /;; at the school level as a function of a school-level variable, specifically a 
dummy variable indicating whether or not the school participated in MCLA. All of the other 
student-level coefficients, except fo, are still assumed constant across schools. In this 
interaction model, fo;, the adjusted mean Spring test score for the control students in school j is 
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expressed as a function of school-level covariates and the MCLA participation variable, and the 
READ 180 treatment effect, 61;, is expressed as a function of MCLA participation: 
P 


Boj =yo+yoWij+ S yo svi +uo; and 


p=l 


Bu =yuotyuWi;, 


where 

W); is an uncentered dummy variable coded 0 for MCLA control schools and 1 for 
MCLA treatment schools; 

Wp+1y is the p™ of P school-level covariates that may be included in the final model 
depending on whether they satisfy criteria for inclusion; 

yoo is the adjusted mean Spring test score for READ 180 control students in MCLA 
control schools; 

yo1 is the adjusted difference between MCLA treatment and control school mean Spring 
test scores (the MCLA treatment effect, controlling for other school-level covariates); 

Yo(p+1) 1s the coefficient for the p" of P school-level covariates that may be included in the 
model depending on whether they satisfy criteria for inclusion; 

Uo; is the unique effect of school 7 and is ~N(0,1); 

vi9 is the READ 180 treatment effect for students in MCLA control schools; and 

11 1s the difference between the READ /80 treatment effects in MCLA treatment and 
control schools. 


The null hypothesis of no interaction between MCLA and READ 180 treatment effects on test 
scores is Ho: 71; = 0 and is tested with a f-statistic. 
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Appendix E: Descriptions of the Baseline Demographic 
Characteristics of Samples of Students for READ 180 Impact 
Analyses in Years 1 through 4 


Equivalence on Student Demographic Characteristics 
Samples for One-Year Impact Analyses 


As described above, 1,740 eligible struggling readers were randomly assigned to the 
READ 180 treatment or control groups in Year 1 of the MSRP study. The grade level and other 
demographic characteristics of these students are presented in Table E-1. 


The number of students decreases as the enrollment grade increases, and this difference is 
reflected more strongly in the control group since approximately equal numbers were randomly 
assigned to the treatment group in each grade. These differences in grade enrollment between 
treatment and control groups emphasize the importance of treating the student’s enrollment grade 
as a covariate in the analyses of READ /80 impact for students from more than one grade. Also, 
all but two students were either African American or Hispanic, which supported the creation of 
two dichotomous covariates to represent membership in these two race/ethnicity groups. Finally, 
the differences in demographic composition of the treatment and control groups were relatively 
minor, although some were statistically significant given the large number of students overall. 
Including these characteristics as student-level covariates in the analytical models helps to 
control for these small differences, as well as reduce the within-school error variance in the 
dependent variables. 


Table E-1: Demographic Characteristics of the Year 1 READ 180 ITT Sample 


Signif. 

Student Characteristic Control* READ 180* Level Total* 
Enrolled in Grade 6 392 37.6% 239 34.2% 631 36.3% 
Enrolled in Grade 7 370 35.5% 233 33.4% 0.05 603 34.7% 
Enrolled in Grade 8 280 26.9% 226 32.4% 506 29.1% 
Female 465 44.6% 286 41.0% 0.13 751 43.2% 
Male 577 55.4% 412 59.0% 989 56.8% 
African American 955 91.6% 657 94.1% 0.05 1612 92.6% 
Hispanic 86 8.2% 40 5.7% 0.05 126 7.2% 
Free or Reduced Lunch 931 89.3% 619 88.7% 0.66 1550 89.1% 
English Language Learner 83 8.0% 34 4.9% 0.01 117 6.7% 
Total 1042 698 1740 


Data sources: MCS enrollment and demographic files, 2006-2007 
*Percentages are based on the total number of students in the control, treatment, or total group. 


In Year 2, 693 eligible struggling readers in sixth grade were randomly assigned to the 
READ 180 treatment or control groups. The demographic characteristics of these students are 
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presented in Table E-2. Other than gender, where the control group has significantly more 
males, the treatment and control groups are quite similar. 


Table E-2: Demographic Characteristics of the Year 2 Sixth-Grade READ 180 ITT Sample 


Signif. 

Student Characteristic Control* READ 180* Level Total* 
Female 169 41.8% 143 49.5% 0.05 312 45.0% 
Male 235 58.2% 146 50.5% 381 55.0% 
African American 382 94.6% 274 94.8% 0.88 656 94.7% 
Hispanic 21 5.2% 15 5.2% 1.00 36 5.2% 
Free or Reduced Lunch 382 94.6% 276 95.5% 0.57 658 94.9% 
English Language Learner 17 4.2% 14 4.8% 0.69 31 4.5% 
Total 404 289 693 


Data sources: MCS enrollment and demographic files, 2007-2008 
a Percentages are based on the total number of students in the control, treatment, or total group. 


In Year 3, 497 eligible struggling readers in sixth grade were randomly assigned to the 
READ 180 treatment or control groups. The demographic characteristics of these students are 
presented in Table E-4. The treatment and control groups are similar in terms of gender and free 
or reduced lunch status composition. The groups differ in terms of race/ethnicity and ELL 
status, however, with the control group having significantly fewer African-American students 
and more Hispanic and ELL students. 


Table E-3: Demographic Characteristics of the Year 3 Sixth-Grade READ 180 ITT Sample 


Signif. 

Student Characteristic Control* READ 180* Level Total* 
Female 102 45.7% 131 47.8% 0.65 233 46.9% 
Male 121 54.3% 143 52.2% 264 53.1% 
African American 204 91.5% 265 96.7% 0.01 469 94.4% 
Hispanic 18 8.1% 9 3.3% 0.02 27 5.4% 
Free or Reduced Lunch 209 93.7% 265 96.7% 0.11 497 95.4% 
English Language Learner 16 7.2% 7 2.6% 0.02 23 4.6% 
Total 223 274 497 


Data sources: MCS enrollment and demographic files, 2008-2009 
*Percentages are based on the total number of students in the control, treatment, or total group. 


In Year 4, 497 eligible struggling readers in sixth grade were randomly assigned to the 
READ 180 treatment or control groups. The demographic characteristics of these students are 
presented in Table E-4. The treatment and control groups are similar in terms of gender and free 
or reduced lunch status composition. The groups differ in terms of race/ethnicity and ELL 
status, however, with the control group having significantly more African-American, Hispanic, 
and ELL students. 
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Table E-4: Demographic Characteristics of the Year 4 Sixth-Grade READ 180 ITT Sample 


Student Characteristic Control” READ 180* Signif. Level Total* 
Female 132 51.0% 118 49.6% 250 50.3% 
Male 127 49.0% 120 50.4% 0.76 247 = 49.7% 
African-American 241 93.1% 232 97.5% 0.02 473 95.2% 
Hispanic 18 6.9% 6 2.5% 0.02 24 4.8% 
Free or Reduced Lunch 256 98.8% 234 98.3% 0.62 490 98.6% 
English Language Learner 20 7.7% 6 2.5% 0.01 26 5.2% 
Total 259 238 497 


Data sources: MCS enrollment and demographic files, 2009-2010 
*Percentages are based on the total number of students in the control, treatment, or total group. 


In Year 4, several analytic samples were created to test the one-year or two-year impact of 
READ 180 on student achievement. Table E-5 presents the demographic characteristics of the 
sample of 2298 sixth grade students from Years 1 through 4. The treatment and control groups 
are similar in terms of gender and free or reduced lunch status composition. The groups differ in 
terms of race/ethnicity and ELL status, however, with the control group having significantly 
more African-American, Hispanic, and ELL students. 


Table E-5: Demographic Characteristics of the Sixth-Grade READ 180 ITT Sample, 


Years 1-4 

Student Characteristic Control” READ 180* Signif. Level Total* 
Female 574 = 45.4% 495 47.9% 1069 46.5% 
Male 690 §=54.6% 539 52.1% 0.24 1229 53.3% 
African-American 1174 92.9% 992 95.9% 0.00 2166 94.3% 
Hispanic 88 7.0% 41 4.0% 0.00 129 5.6% 
Free or Reduced Lunch 1193 944% 982 95.0% 0.53 2175 94.6% 
English Language Learner 81 6.4% 35 3.4% 0.00 116 5.0% 
Total 1264 1034 2298 


Data sources: MCS enrollment and demographic files, 2006-2007 through 2009-2010 

*Percentages are based on the total number of students in the control, treatment, or total group. 

Note: The current sample size is slightly smaller than the cumulative sample size of the past four years because 
students who were retained in grade have been removed from the current analyses. 


Table E-6 presents the demographic characteristics of the sample of 3407 students in sixth, 
seventh, and eighth grades in Years | through 4 with one year of treatment. The treatment and 
control groups are similar in terms of gender and free or reduced lunch status composition. The 
groups differ in terms of race/ethnicity and ELL status, however, with the control group having 
significantly more African-American, Hispanic, and ELL students. 
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Table E-6: Demographic Characteristics of the READ 180 ITT Sample Students with One 
Year of Treatment All Grades, All Years 


Student Characteristic Control* READ 180* Signif. Level Total* 
Female 868 45.4% 677 = 45.3% 1545 45.3% 
Male 1046 54.6% 816 54.7% 1.00 1862 54.7% 
African-American 1768 92.4% | 1422 95.2% 0.00 3190 93.6% 
Hispanic 143 7.5% 70 4.7% 0.00 213 6.3% 
Free or Reduced Lunch 1765 92.2% | 1388 93.0% 0.41 3153 = 92.5% 
English Language Learner 136 7.1% 61 4.1% 0.00 197 5.8% 
Total 1914 1493 3407 


Data sources: MCS enrollment and demographic files, 2006-2007 through 2009-2010 

*Percentages are based on the total number of students in the control, treatment, or total group. 

Note: The current sample size is slightly smaller than the cumulative sample size of the past four years because 
students who were retained in grade have been removed from the current analyses. 


Samples for Two-Year Impact Analyses 


In Year 2, 820 students in the seventh and eighth grades remained enrolled in a participating 
school out of the 1,234 eligible struggling readers in the sixth and seventh grades in the Year 1 
READ I80 ITT group. The demographic characteristics of these “stayers,” compared with the 
414 “leavers” who were not in a participating school on May 13, 2008, are presented in Table E- 
re 


Table E-7: Demographic Characteristics of the Year 2 “Stayers” and “Leavers” from the 
Year 1 READ 180 ITT Sample 


Control* Read 180* 
Student Signif. Signif. 
Characteristic “Stayers” “Leavers” Level “Stayers” “Leavers” Level 
aa in Grade | 569 51.90% | 132 -50.60% 160 -50.20% 179 -51.60% 

. 0.73 0.76 
Suis in Grade |o44 48.10% | 129 -49.40% 159 -49.80% 174 -48.40% 
Female 236 -47.10% |118 -45.20% |. [139 -43.60% [61 -39.90% | 
Male 265 -52.90% |143 -54.80% | ~ 180 -56.40% 192 -60.10% 
African American | 457 -91.20% | 238 -91.20% | 0.89 |297 -93.10%|144 -94.10% | 0.68 
Hispanic 43 -8.60% |22 -840% | 1.00 ]21 -660% 19 -5.90% | 0.76 
cher Reduced | 449 87.80% |240 -92.00% | 0.08 |287 -90.00% |137  -89.50% | 0.37 
eet as 46 -9.20% |19 -7.30% | 0.37118 -5.60% 17 (%) 0.63 
Total 501 261 319 153 


Data sources: MCS enrollment and demographic files, 2006-2007 and 2007-2008 
*Percentages are based on the total for the control and treatment groups for each type of student. 


Appendix E. MSRP Evaluation Report, Y4 March 1, 2011. Page 5 of 6 


In Year 3, 475 seventh-grade students remained enrolled in a participating school out of the 
693 eligible struggling readers in the sixth grade in the Year 2 READ ISO ITT group. The 
demographic characteristics of these “stayers,” compared with the 140 “leavers” who were not in 
a participating school on May 6, 2009, are presented in Table E-8. 


Differences between treatment and control groups for the 475 seventh-grade students in Year 
3 who remained in the READ 180 ITT sample are not statistically significant. The largest 
disparity between the groups is in the gender composition, but even these differences have a 
significance level greater than 0.05. Among the 140 “leavers”, there are no significant 
differences between the treatment and control groups’ demographic characteristics. 


Table E-8: Demographic Characteristics of the Year 3 Seventh-Grade “Stayers” and 
“Leavers” from the Year 2 READ 180 ITT Sample 


Control* Read 180* 

Student Signif. Signif. 
Characteristic “Stayers” “Leavers” Level “Stayers” “Leavers” Level 
Female 127 45.50%] 31 36.50% 644 105 53.60% | 21 38.20% B04 
Male 152 54.50%] 54 63.50% 91 46.40% | 34 61.80% 

African American | 264 94.60% | 81 95.30% | 0.81 | 185 94.40% | 55 100% 0.07 
Hispanic 15 5.40% 4 4.70% | 0.81 11 5.60% 0 0% 0.07 
ats Reduced |562 93.90% | 80 94.10% | 0.94 }189 96.40%] 50 90.90% | 0.09 


English Language 
Learner 
Total 279 85 196 55 


Data sources: MCS enrollment and demographic files, 2007-2008 through 2008-2009 
*Percentages are based on the total for the control and treatment groups for each type of student. 


13 4.70% 3 3.50% | 0.66 | 11 5.60% 0 0% 0.07 


Table E-9 presents the demographic characteristics of the sample of 312 seventh grade 
“stayers” in Year 4 with two years of treatment. Stayers are students who were in a participating 
school on May 7, 2010. The treatment and control groups do not differ significantly on any of 
these demographic characteristics. 
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Table E-9: Demographic Characteristics of the Year 4 Seventh-Grade “Stayers” in 
READ 180 ITT Sample 


Student Characteristic Control* READ 180* Signif. Level Total* 
Female 70 49.6% 83 48.5% 153. 49.0% 
Male 71 ~=—50.4% 88 51.5% 0.846 159 51.0% 
African-American 130 92.2% 164 95.9% 0.162 294 94.2% 
Hispanic 10 7.1% 7 4.1% 0.245 17 5.4% 
Free or Reduced Lunch 131 92.9% 162 94.7% 0.501 293 93.9% 
English Language Learner 7 5.0% 5 2.9% 0.351 12 3.8% 
Total 141 171 312 


Data sources: MCS enrollment and demographic files, 2008-2009 and 2009-2010 
*Percentages are based on the total number of students in the control, treatment, or total group. 


Lastly, Table E-10 presents the demographic characteristics of the sample of 1601 seventh 
and eighth grade “stayers” in Years 1 through 4 with two years of treatment. The treatment and 
control groups do not differ significantly, except that the control group has significantly more 
ELL students than the treatment group. 


Table E-10: Demographic Characteristics of the Seventh- and Eighth-Grade “Stayers” in 
READ 180 ITT Sample Students with Two Years of Treatment, All Years 


Student Characteristic Control* READ 180* Signif. Level Total* 
Female 430 46.9% 327 = 47.7% 757 = 47.3% 
Male 486 53.1% 358 52.3% 0.753 844 52.7% 
African-American 846 §=92.4% 646 94.3% 0.126 1492 93.2% 
Hispanic 68 TA% 38 5.5% 0.135 106 6.6% 
Free or Reduced Lunch 829 90.5% 637 93.0% 0.076 1466 91.6% 
English Language Learner 66 7.2% 33 4.8% 0.05 99 6.2% 
Total 916 685 1601 


Data sources: MCS enrollment and demographic files, 2006-2007 through 2009-2010 

*Percentages are based on the total number of students in the control, treatment, or total group. 

Note: The current sample size is slightly smaller than the cumulative sample size of the past four years because 
students who were retained in grade have been removed from the current analyses. 
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Appendix F: Descriptions of the Baseline Achievement 
Characteristics of Samples of Students for READ 180 Impact 
Analyses in Years 1 through 4 


Equivalence on Baseline Achievement 


Year 1. Comparisons between treatment and control groups on the baseline 2006 ITBS and 
TCAP test scores were carried out for the 1,740 students in the Year 1 ITT sample and the 820 
Year 2 “stayers.” Treatment and control comparisons were made on the baseline 2007 ITBS 
and TCAP test scores for the Year 2 sixth-grade ITT sample and on the baseline 2008 ITBS and 
TCAP test scores for the Year 3 sixth-grade ITT sample. 


Table 30 describes the differences between Year 1 READ /80 treatment and control groups 
on baseline 2006 test scores for the three ITBS standard scores and the four TCAP content area 
assessments. With random assignment, the treatment and control groups should be very similar 
on all seven test scores. 


The treatment group performed higher on all seven tests. However, the significance level for 
each estimated difference is greater than 0.05, allowing one to conclude that the treatment and 
control groups are statistically equivalent in terms of their baseline 2006 ITBS and TCAP test 
scores. 
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Table F-1: Comparison of Year I READ 180 Treatment and Control Groups on Baseline 
2006 Scores on Each Achievement Test 


Means Signif. 

Test Score Control READ 180 Level 

ITBS Total Reading 190.2 191.5 0.097 
Standard Score (940)* (656) : 

ITBS Comprehension 186.2 188.0 0.059 
Standard Score (944) (660) : 

ITBS Vocabulary 194.1 195.1 0.354 
Standard Score (950) (658) : 

TCAP Reading/LA 474.8 476.5 0.188 
Scale Score (1042) (698) ; 

TCAP Mathematics 484.4 487.4 0.062 
Scale Score (1040) (697) ; 

TCAP Science 179.1 180.5 0.116 
Scale Score (1006) (686) ; 

TCAP Social Studies 184.0 184.5 0.593 
Scale Score (1007) (685) : 


Data sources: ITBS and TCAP, 2006 
*Numbers in parentheses are the numbers of students in each group having a valid test score. 


Year 2. Table 31 describes the differences between Year 2 sixth-grade READ 1/80 treatment and 
control groups on baseline 2007 tests. Again, with random assignment, the 693 students in the 


treatment and control groups should be very similar on all seven test scores. 


Table F-2: Comparison of Year 2 Sixth-Grade READ 180 Treatment and Control Groups on 
Baseline 2007 Scores on Each Achievement Test 


Means Signif. 
Test Score Control READ 180 Level 
ITBS Total Reading 184.9 182.0 0.007 
Standard Score (364)* (247) ; 
ITBS Comprehension 182.8 179.7 0.014 
Standard Score (365) (247) : 
ITBS Vocabulary 186.9 184.7 0.104 
Standard Score (365) (252) . 
TCAP Reading/LA 487.8 482.4 0.004 
Scale Score (403) (289) ; 
TCAP Mathematics 489.8 487.3 0.208 
Scale Score (403) (289) ; 
TCAP Science 184.4 182.8 0.204 
Scale Score (400) (283) , 
TCAP Social Studies 189.5 188.9 0.551 
Scale Score (399) (283) ; 


Data sources: ITBS and TCAP, 2007 
*Numbers in parentheses are the numbers of students in each group having a valid test score. 
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The control group performed higher on all seven test scores. This advantage was statistically 
significant for the ITBS Comprehension and Total Reading standard scores and for the TCAP 
Reading/LA scale score. Thus, even though students were assigned to treatment and control 
groups randomly, the control group scored significantly higher on most of the baseline reading 
measures. Treating the 2007 test scores as covariates in the analyses of the impact of READ 180 
on 2008 test scores allows some adjustment to be made for these differences. 


Year 3. Table 32 describes the differences between Year 3 sixth-grade READ 180 treatment and 
control groups on baseline 2008 test scores. The two groups have the same initial achievement 
level; there are no statistically significant differences in their baselines ITBS and TCAP test 
scores. 


Table F-3: Comparison of Year 3 Sixth-Grade READ 180 Treatment and Control Groups on 
Baseline 2008 Scores on Each Achievement Test 


Means Signif. 
Test Score Control READ 180 Level 
ITBS Total Reading 183.0 183.1 
Standard Score (205)* (237) 0.886 
ITBS Comprehension 178.6 179.0 
Standard Score (215) (263) 0.827 
ITBS Vocabulary 187.5 187.3 
Standard Score (205) (238) 0.929 
TCAP Reading/LA 482.2 482.1 
Scale Score (222) (273) 0.950 
TCAP Mathematics 482.8 482.2 
Scale Score (222) (273) 0.804 
TCAP Science 180.2 180.6 
Scale Score (222) (273) 0.762 
TCAP Social Studies 187.3 187.1 
Scale Score (219) (273) 0.873 


Data sources: ITBS and TCAP, 2008 
*Numbers in parentheses are the numbers of students in each group having a valid test score. 


Year 4. Table 33 describes the differences between Year 4 sixth-grade READ 180 treatment and 
control groups on baseline 2009 test scores. The two groups have the same initial achievement 
level; there are no statistically significant differences in their baselines ITBS and TCAP test 
scores. 
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Table F-4: Comparison of Year 4 Sixth-Grade READ 180 Treatment and Control Groups on 
Baseline 2009 Scores on Each Achievement Test 


Means Signif. 
Test Score Control READ 180 Level 
ITBS Total Reading 23.75 22.09 
NCE Score (242)* (215) 0.103 
ITBS Comprehension 25.46 25.12 
NCE Score (242) (216) 0.734 
ITBS Vocabulary 24.89 22.67 
NCE Score (243) (215) 0.057 
TCAP Reading/LA Scale Score 475.0 472.8 
(259) (238) 0.297 


Data source: ITBS and TCAP, 2009 


*Numbers in parentheses are the numbers of students in each group having a valid test score. 
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Appendix G: Complete Results of Multi-Level Analyses of 
READ 180 One-Year and Two-Year Impacts on Student 
Achievement in Year 1 through Year 4 


Table G-1: Final Model for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Total Reading Scores of Students in All Grades in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 192.60 0.55 4 349.13 0.000 
Percent African American 0.95 0.52 4 1.82 0.142 
Percent Female 0.64 0.35 4 1.84 0.138 
Percent ELL 1.81 0.77 4 2.34 0.076 
Student | Grade 7 5.64 0.94 1214 5.98 0.000 
Grade 8 12.37 1.07 1214 11.54 0.000 
READ 180 -0.50 0.81 1214 -0.63 0.532 
ELL -3.62 1.66 1214 -2.17 0.030 
ITBS Total Reading 06 0.04 0.03 1214 16.11 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.17 0.005 
Student | Level 1 185.34 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-2: Final Model for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Comprehension Scores of Students in All Grades in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 186.99 0.72 3 260.55 0.000 
Percent African American 1.89 0.68 3 2.77 0.068 
Percent SPED -0.71 0.22 3 -3.22 0.070 
Percent ELL 2.12 0.91 3 2.33 0.091 
Size 0.01 0.00 3 2.15 0.111 
Student | Grade 7 7.15 1.16 1223 6.19 0.000 
Grade 8 10.90 1.28 1223 8.54 0.000 
READ 180 -0.03 1.03 1223 -0.03 0.976 
Gender 3.69 0.99 1223 3.74 0.000 
Hispanic 5.41 3.81 1223 1.42 0.155 
ELL -7.75 3.85 1223 -2.01 0.044 
ITBS Comprehension 06 0.37 0.03 1223 12.99 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.22 0.008 
Student | Level 1 293.34 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-3: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Vocabulary Scores of Students in All Grades in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 197.47 0.67 6 293.73 0.000 
Percent ELL 0.48 0.16 6 3.05 0.024 
Student | Grade 7 5.25 1.22 1237 4.31 0.000 
Grade 8 15.97 1.36 1237 11.75 0.000 
READ 180 0.08 1.02 1237 0.08 0.937 
Gender -2.10 1.01 1237 -2.08 0.037 
Hispanic -4.71 2.10 1237 -2.24 0.025 
ITBS Vocabulary 06 0.35 0.03 1237 13.05 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.02 0.003 
Student | Level 1 310.37 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-4: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 2007 TCAP Reading/LA Scores of Students in All Grades in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 496.76 0.86 4 574.83 0.000 
Percent African American 2.21 0.80 4 2.75 0.051 
Percent Female 2.49 0.54 4 4.64 0.009 
Percent ELL 3.71 1.18 4 3.14 0.042 
Student | Grade 7 8.79 1.51 1624 5.83 0.000 
Grade 8 13.42 1.62 1624 8.28 0.000 
READ 180 0.36 1.31 1624 0.28 0.781 
Gender 3.89 1.28 1624 3.03 0.003 
Hispanic 8.51 4.93 1624 1.72 0.084 
FRL -5.44 2.04 1624 -2.66 0.008 
ELL -15.17 5.07 1624 -2.99 0.003 
TCAP Reading 06 0.33 0.03 1624 12.88 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.17 0.012 
Student | Level 1 658.24 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-5: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Mathematics Scores of Students in All Grades in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 500.05 1.26 5 397.76 0.000 
Percent Female -1.12 0.72 5 -1.56 0.179 
Percent FRL -1.05 0.47 5 -2.24 0.074 
Student | Grade 7 2.77 1.62 1621 1.71 0.088 
Grade 8 8.00 1.72 1621 4.63 0.000 
READ 180 0.17 1.42 1621 0.12 0.904 
Gender 5.52 1.38 1621 4.01 0.000 
African American 67.13 19.51 1621 3.44 0.001 
Hispanic 77.96 20.17 1621 3.87 0.000 
ELL -7.82 5.44 1621 -1.44 0.150 
TCAP Math 06 0.51 0.02 1621 24.09 0.000 
Random Effects 
Level Variance Components Variance Icc* 
School Level 2 5.65 0.021 
Student | Level 1 754.78 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 


Table G-6: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Science Scores of Students in All Grades in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 185.62 0.57 5 324.04 0.000 
Percent SPED -0.25 0.12 5 -2.00 0.100 
Size -0.01 0.00 5 -4.43 0.007 
Student | Grade 7 -2.42 1.65 1551 -1.47 0.143 
READ 180 -0.48 0.76 1551 -0.63 0.528 
ELL -5.19 1.15 1551 -4.53 0.000 
TCAP Science 06 0.22 0.01 1551 15.57 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.03 0.008 
Student | Level 1 258.61 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 
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Table G-7: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Social Studies Scores of Students in All Grades in Year 1 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 185.16 1.24 7 149.02 0.000 
Student | READ 180 0.81 0.67 1545 1.21 0.226 

African American 3.02 1.41 1545 2.14 0.033 
FRL -1.36 0.73 1545 -1.85 0.064 
TCAP Social Studies 06 0.22 0.04 1545 5.68 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 7.98 0.034 
Student | Level 1 246.20 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-8: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Total Reading Scores of Sixth-Graders in Year 1 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 183.78 0.65 7 282.77 0.000 
Student | READ 180 0.45 1.05 460 0.43 0.665 

FRL -3.06 1.64 460 -1.87 0.061 
ELL -5.61 2.15 460 -2.61 0.010 
ITBS Total Reading 06 0.53 0.04 460 13.08 0.000 
Random Effects 
Level Variance Components Variance Icc* 
School Level 2 .03 .000 
Student | Level 1 164.64 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-9: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Comprehension Scores of Sixth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 180.32 0.83 7 218.70 0.000 
Student | READ 180 -1.03 1.34 460 -0.77 0.441 
Gender 4.35 1.30 460 3.32 0.001 
African American -10.18 5.78 460 -1.76 0.079 
FRL -3.76 2.09 460 -1.80 0.071 
ELL -15.19 5.98 460 -2.54 0.012 
ITBS Comprehension 06 0.42 0.04 460 9.43 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.82 0.003 
Student | Level 1 234.99 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-10: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Vocabulary Scores of Sixth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 186.81 0.88 6 211.51 0.000 
Percent ELL 0.49 0.51 6 2.38 0.054 
Student | READ 180 2.68 1.40 464 1.92 0.056 
Gender -2.47 1.37 464 -1.80 0.072 
ELL -7.20 2.90 464 -2.49 0.013 
ITBS Vocabulary 06 0.39 0.04 464 9.83 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 3.24 0.012 
Student | Level 1 263.97 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-11: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Reading/LA Scores of Sixth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 488.88 1.37 4 357.45 0.000 
Percent African American 2.69 1.30 4 2.07 0.104 
Percent Female 2.95 0.85 4 3.47 0.037 
Percent ELL 3.86 1.92 4 2.01 0.112 
Student | READ 180 0.01 2.17 590 0.01 0.996 
Gender 4.99 2.09 590 2.39 0.017 
African American -15.14 8.30 590 -1.82 0.068 
FRL -7.26 3.54 590 -2.05 0.040 
ELL -17.42 8.93 590 -1.95 0.051 
TCAP Reading 06 0.33 0.04 590 7.49 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 22.40 0.030 
Student | Level 1 718.44 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-12: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Mathematics Scores of Sixth-Graders in Year 1 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 492.27 1.52 5 323.93 0.000 
Percent African American 0.58 0.33 5 1.72 0.145 
Size 0.03 0.01 5 3.44 0.024 
Student | READ 180 2.38 2.19 592 1.09 0.279 
Gender 4.63 2.09 592 2.21 0.027 
African American -5.93 4.48 592 -1.33 0.186 
TCAP Math 06 0.53 0.04 592 13.55 0.000 
Random Effects 
Level Variance Components Variance Icc* 
School Level 2 15.02 0.018 
Student | Level 1 839.38 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-13: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 


Impact of READ 180 on 2007 TCAP Science Scores of Sixth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 186.55 85 5 220.66 0.000 
Percent Female -1.01 0.50 5 -2.01 0.100 
Percent ELL -0.72 0.26 5 -2.82 0.038 
Student | READ 180 1.04 1.30 565 0.80 0.423 
Gender -1.71 1.27 565 -1.35 0.178 
African American -8.75 5.84 565 -1.50 0.135 
ELL -9.85 7.37 565 -1.34 0.182 
TCAP Science 06 0.26 0.04 565 6.47 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 3.00 0.012 
Student | Level 1 243.54 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 


Table G-14: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Social Studies Scores of Sixth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 186.67 1.42 6 131.07 0.000 
Percent ELL -0.62 0.39 6 -1.60 0.159 
Student | READ 180 0.76 1.51 567 0.50 0.615 
Social Studies 06 0.30 0.05 567 6.08 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 17.37 0.052 
Student | Level 1 314.03 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 
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Table G-15: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Total Reading Scores of Sixth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 193.57 .90 6 214.91 0.000 
Percent SPED -38.52 0.23 6 -1.71 0.138 
Student | READ 180 -2.26 1.37 419 -1.65 0.099 
ITBS Reading Total 06 0.42 0.04 419 10.10 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 3.25 0.014 
Student | Level 1 230.90 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-16: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Comprehension Scores of Seventh-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 188.92 1.18 3 159.53 0.000 
Percent African American 3.75 1.13 3 3.30 0.071 
Percent SPED -1.23 0.37 3 -3.31 0.071 
Percent ELL 4.10 1.53 3 2.69 0.070 
Size 0.01 0.01 3 1.73 0.179 
Student | READ 180 -1.56 1.80 421 -0.87 0.384 
Gender 2.70 1.74 421 1.55 0.121 
ITBS Comprehension 06 0.35 0.05 421 7.43 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 8.83 0.025 
Student | Level 1 345.99 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-17: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Vocabulary Scores of Seventh-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 197.79 1.14 6 173.11 0.000 
Percent African American -0.27 0.18 6 -1.50 0.184 
Student | READ 180 -2.07 1.77 428 -1.17 0.244 
ITBS Vocabulary 06 0.36 04 428 8.61 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 1.53 0.004 
Student | Level 1 377.19 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-18: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Reading/LA Scores of Seventh-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 497.91 1.38 6 361.50 0.000 
Percent Female 1.19 0.73 6 1.64 0.152 
Student | READ 180 0.27 1.94 564 0.14 0.890 
ELL -7.90 3.57 564 -2.21 0.027 
TCAP Reading 06 0.36 0.04 564 9.46 0.000 
Random Effects 
Level Variance Components Variance Icc* 
School Level 2 7.80 0.013 
Student | Level 1 599.19 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-19: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Mathematics Scores of Seventh-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 502.33 3.00 6 167.31 0.000 
Percent FRL -2.44 1.20 6 -2.02 0.088 
Student | READ 180 -3.24 2.36 563 -1.37 0.171 
Gender 6.15 2.28 563 2.70 0.008 
TCAP Math 06 0.58 0.04 563 15.46 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 79.05 0.072 
Student | Level 1 1024.75 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-20: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Science Scores of Seventh-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 183.84 1.03 6 178.53 0.000 
Percent FRL -1.01 0.34 6 -2.99 0.026 
Student | READ 180 -2.62 1.59 539 -1.64 0.101 
FRL 4.22 2.49 539 1.70 0.090 
ELL -8.29 3.61 539 -2.29 0.022 
TCAP Science 06 0.22 0.04 539 4.96 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 1.95 0.005 
Student | Level 1 357.42 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-21: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Social Studies Scores of Seventh-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 183.58 1.08 6 169.89 0.000 
Percent FRL -1.38 0.40 6 -3.48 0.016 
Student | READ 180 1.58 1.30 535 1.22 0.225 
TCAP Social Studies 06 0.15 0.03 535 4.83 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 13.13 0.054 
Student | Level 1 229.26 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-22: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Total Reading Scores of Eighth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 202.59 1.32 5 153.95 0.000 
Percent Female 1.46 0.79 5 1.86 122 
Percent ELL 1.31 0.39 5 3.34 0.026 
Student | READ 180 0.54 1.81 329 0.30 0.766 
ELL -8.99 4.36 329 -2.06 0.040 
ITBS Total Reading 06 0.36 0.06 329 6.19 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 8.75 0.029 
Student | Level 1 293.19 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-23: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Comprehension Scores of Eighth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 194.54 1.54 4 126.50 0.000 
Percent Female 176.89 0.93 4 1.90 0.128 
Percent ELL 154.63 0.46 4 3.34 0.039 
Percent FRL -88.74 0.48 4 -1.86 0.134 
Student | READ 180 1.17 2.17 330 0.54 0.589 
African American 31.47 19.82 330 1.59 0.113 
Hispanic 45.71 21.12 330 2.16 0.031 
ELL -23.20 8.27 330 -2.80 0.006 
ITBS Comprehension 06 0.35 0.06 330 6.37 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 19.73 0.044 
Student | Level 1 432.82 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-24: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 ITBS Vocabulary Scores of Eighth-Graders in Year 1 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 211.17 1.58 4 133.99 0.000 
Percent SPED 2.11 92.74 4 2.28 0.081 
Percent ELL 1.29 51.29 4 2.52 0.063 
Percent FRL -1.93 98.82 4 -1.95 0.120 

Student | READ 180 -0.74 2.26 337 -0.33 0.742 
ITBS Vocabulary 06 0.29 0.06 337 4.82 0.000 

Random Effects 

Level Variance Components Variance icc’ 

School Level 2 14 0.000 

Student | Level 1 454.56 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-25: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Reading/LA Scores of Eighth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 507.71 2.17 4 233.63 0.000 
Percent Female 2.48 1.17 4 2.11 0.099 
Percent ELL 2.44 0.73 4 3.35 0.039 
Size -0.03 0.01 4 -2.86 0.048 
Student | READ 180 -1.80 2.99 456 -0.60 0.546 
Gender 6.90 2.77 456 2.50 0.013 
African American 39.87 29.65 456 1.35 0.179 
Hispanic 55.27 31.29 456 1.77 0.078 
FRL -6.54 4.17 456 -1.57 0.117 
ELL -25.36 11.33 456 -2.24 0.026 
TCAP Reading 06 0.31 0.05 456 5.74 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 10.80 0.011 
Student | Level 1 944.31 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 


Table G-26: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Mathematics Scores of Eighth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 508.93 2.10 3 242.83 0.000 
Percent SPED 2.67 1.15 3 2.31 0.093 
Percent ELL 3.13 0.75 3 4.17 0.045 
Percent FRL -2.86 1.23 3 -2.34 0.091 
Size -0.03 0.01 3 -3.38 0.071 
Student | READ 180 -1.03 2.90 454 -0.36 0.721 
Gender 3.60 2.69 454 1.34 0.181 
African American 170.34 28.66 454 5.94 0.000 
Hispanic 189.46 30.31 454 6.25 0.000 
ELL -16.57 11.11 454 -1.49 0.136 
TCAP Math 06 0.45 0.03 454 13.85 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 53.29 0.042 
Student | Level 1 1219.24 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 
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Table G-27: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Science Scores of Eighth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 188.01 1.02 6 184.62 0.000 
Size -0.02 0.00 6 -6.79 .000 
Student | READ 180 -1.68 1.41 435 -1.19 0.234 
Gender -2.06 1.33 435 -1.55 0.121 
TCAP Science 06 0.18 0.04 435 5.01 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 25.91 0.115 
Student | Level 1 200.01 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-28: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2007 TCAP Social Studies Scores of Eighth-Graders in Year 1 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 184.92 1.64 7 112.54 0.000 
Student | READ 180 0.05 1.47 435 0.03 0.976 
African American 27.48 14.47 435 1.90 0.058 
Hispanic 20.53 15.03 435 1.37 0.173 
TCAP Social Studies 06 0.30 0.05 435 6.26 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 14.24 0.060 
Student | Level 1 224.56 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-29: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2008 ITBS Total Reading Scores of Sixth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 184.43 0.65 6 283.34 0.000 
Percent SPED -0.54 0.17 6 -3.16 0.022 
Student | READ 180 -0.73 1.01 495 -0.73 0.468 
ITBS Total Reading 07 0.45 0.04 495 11.36 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.01 0.017 
Student | Level 1 119.17 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-30: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2008 ITBS Comprehension Scores of Sixth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 181.85 0.91 7 199.87 0.000 
Student | READ 180 -1.83 1.33 499 -1.38 0.017 
Gender 2.35 1.29 499 1.82 0.069 
ITBS Comprehension 07 0.40 0.04 499 9.56 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.77 0.005 
Student | Level 1 207.15 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-31: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2008 ITBS Vocabulary Scores of Sixth-Graders in Year 2 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 187.43 0.85 6 221.04 0.000 
Percent SPED -0.82 0.22 6 -3.71 0.013 
Student | READ 180 -0.61 1.31 504 -0.47 0.639 
FRL 4.81 2.88 504 1.67 0.096 
ITBS Vocabulary 07 0.33 0.04 504 8.36 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.01 0.012 
Student | Level 1 204.71 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-32: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2008 TCAP Reading/LA Scores of Sixth-Graders in Year 2 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 4494.66 1.61 5 306.47 0.000 
Percent SPED 0.99 0.42 5 2.36 0.063 
Enrollment 0.02 0.01 5 2.93 0.034 
Student | READ 180 1.89 2.28 662 0.83 0.407 
Gender 6.93 2.16 662 3.21 0.002 
TCAP Reading 07 0.48 0.05 662 10.41 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 1.33 0.003 
Student | Level 1 758.82 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-33: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2008 TCAP Mathematics Scores of Sixth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 499.39 2.12 6 235.95 0.000 
Percent SPED 1.76 0.60 6 2.92 0.028 
Student | READ 180 -3.12 1.98 664 -1.58 0.115 
TCAP Math 07 0.63 0.04 664 16.16 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 21.21 0.035 
Student | Level 1 593.54 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-34: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2008 TCAP Science Scores of Sixth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 186.55 0.77 6 240.89 0.000 
Percent Female -0.59 0.35 6 -1.67 0.146 
Student | READ 180 -0.18 1.18 654 -0.16 0.876 
Gender -1.60 1.16 654 -1.38 0.169 
TCAP Science 07 0.33 0.03 654 9.60 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.15 0.001 
Student | Level 1 217.48 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-35: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on 2008 TCAP Social Studies Scores of Sixth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 187.77 1.14 3 162.66 0.000 
Percent African American -3.52 1.21 3 -2.91 0.067 
Percent SPED 2.13 0.76 3 2.81 0.067 
Percent ELL -3.99 1.65 3 -2.42 0.084 
Percent FRL -1.76 0.79 3 -2.22 0.102 
Student | READ 180 -2.35 1.37 642 -1.71 0.087 
Gender -2.83 1.32 642 2.14 0.032 
African American -14.02 5.44 642 -2.58 0.010 
ELL -12.22 5.85 642 -2.09 0.037 
TCAP Social Studies 07 0.41 0.05 642 8.63 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 3.45 0.031 
Student | Level 1 275.22 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-36: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on Spring 2009 ITBS Reading Total Scores of Sixth-Graders in Year 3 


Fixed Effects 


Level Effect Estimate St Err DF t-value Pr > |t| 
School Intercept 184.22 0.92 4 200.27 0.000 
Percent Female 0.98 0.44 4 2.21 0.089 
ey, manguage 0.39 0.22 4 1.80 0.144 
Student READ 180 -2.04 1.17 374 -1.75 0.081 
Gender -1.80 1.12 374 -1.61 0.108 
ITBS Reading Total 08 0.45 0.05 374 9.89 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School School intercepts 0.04 0.001 
Student Residual 118.65 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-37: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on Spring 2009 ITBS Comprehension Scores of Sixth-Graders in Year 3 


Fixed Effects 
Level Effect Estimate St Error DF t-value Pr > |t| 
School Intercept 179.65 1.03 7 173.61 0.000 
Student READ 180 -1.41 1.40 407 -1.01 0.314 
English Language Learner -9.51 6.86 407 -1.39 0.166 
African American -8.47 6.38 407 -1.33 0.185 
ITBS Comprehension 08 0.38 0.05 407 8.27 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School School intercepts 0.02 0.000 
Student Residual 198.43 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 


Table G-38: Final Model* for Estimating Fixed and Random Effects for Determining the 


One-Year Impact of READ 180 on Spring 2009 ITBS Vocabulary Scores of Sixth-Graders in 


Year 3 
Fixed Effects 

Level Effect Estimate St Err DF t-value Pr > |t| 

School Intercept 188.92 1.15 4 164.16 0.000 
Percent Female 1.34 0.55 4 2.41 0.07 
panei Fanguage 0.51 0.27 4 1.89 0.129 

Student READ 180 -2.34 1.46 375 -1.60 0.111 
Gender -3.80 1.41 375 -2.70 0.008 
ITBS Vocab 08 0.35 0.05 375 7.61 0.000 

Random Effects 

Level Variance Components Variance icc’ 

School School intercepts 0.02 0.003 

Student Residual 186.46 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-39: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on Spring 2009 TCAP Reading Scores of Sixth-Graders in Year 3 


Fixed Effects 
Level Effect Estimate St Err DF t-value Pr > |t| 
School Intercept 473.76 3.85 6 122.94 0.000 
School Size 0.02 0.02 6 1.45 0.197 
Student READ 180 6.95 3.20 479 2.17 0.030 
Gender 5.62 3.02 479 1.86 0.063 
African American -23.07 6.94 479 -3.32 0.001 
TCAP Reading 08 0.55 0.06 479 8.91 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School School intercepts 66.44 0.053 
Student Residual 1086.14 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 


Table G-40: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 


Impact of READ 180 on Spring 2009 TCAP Math Scores of Sixth-Graders in Year 3 


Fixed Effects 
Level Effect Estimate St Err DF t-value Pr > |t| 
School Intercept 491.72 2.36 5 208.54 0.000 
Percent African American 4.69 2.05 5 2.28 0.070 
ia heat nenguege 6.75 2.94 5 2.29 0.069 
Student READ 180 -3.21 2.58 480 -1.24 0.215 
TCAP Math 08 0.53 0.05 480 11.02 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School School intercepts 10.83 0.016 
Student Residual 717.76 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Impact of READ 180 on Spring 2009 TCAP Science Scores of Sixth-Graders in Year 3 


Level Effect Estimate St Err DF t-value Pr > |t| 

School Intercept 183.31 1.36 5 134.33 0.000 
Percent African American 2.18 1.23 5 1.77 0.136 
eee panyuade 3.43 1.76 5 1.94 0.108 

Student READ 180 -0.87 1.45 474 -0.60 0.550 
Gender -4.41 1.38 474 -3.20 0.002 
English Language Learner -9.96 6.32 474 -1.58 0.12 
African American -10.48 5.78 474 -1.81 0.07 
Free/Reduced Lunch -7.74 3.31 474 -2.34 0.02 
TCAP Science 08 0.26 0.04 474 5.94 0.00 

Random Effects 

Level Variance Components Variance icc’ 

School School intercepts 4.26 0.042 

Student Residual 224.79 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-42: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on Spring 2009 TCAP Social Studies Scores of Sixth-Graders in Year 3 


Fixed Effects 
Level Effect Estimate St Err DF t-value Pr > |t| 
School Intercept 183.64 1.24 5 147.61 0.000 
Percent African American -0.41 0.16 5 -2.48 0.055 
a ers 0.57 0.35 5 -1.60 | 0.169 
Student READ 180 -0.58 1.58 476 -0.37 0.714 
TCAP Social Studies 08 0.37 0.06 476 6.10 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School School intercepts 0.02 0.009 
Student Residual 271.45 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Appendix G. MSRP Evaluation Report, Year 4 March 1, 2011. Page 23 of 44 


Table G-43: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on ITBS Total Reading Scores of Sixth-Graders in Year 4 


Fixed Effects 
Level Effect Estimate St Err DF | t-value Pr > |t| 
School Intercept 18.83 0.77 5 24.59 0.000 
Percent Female 0.80 0.43 5 1.86 0.121 
Percent ELL 0.53 0.20 5 2.66 0.044 
Student READ 180 0.51 1.03 376 0.49 0.621 
African American 6.26 2.37 376 2.64 0.009 
ITBS Total Reading Pre 0.39 0.05 376 8.25 0.000 
Random Effects 
Level Variance Components Variance 
School School intercepts 0.14 
Student Residual 93.85 
Random Effects (From Unconditional Model) 
Level Variance Components Variance icc’ 
School School intercepts 0.29 0.002 
Student Residual 116.59 


* Final model includes the treatment effect of READ 180 and all covariates with p<0.2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 


Table G-44: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on ITBS Comprehension Scores of Sixth-Graders in Year 4 


Fixed Effects 
Level Effect Estimate St Err DF | t-value | Pr> |t| 
School Intercept 21.07 0.86 3 | 24.39 0.000 
Percent African American 1.10 0.67 3 1.64 0.198 
Percent Female 0.94 0.50 3 1.87 0.154 
Percent ELL 1.61 0.98 3 1.65 0.195 
Size 0.01 0.00 3 2.24 0.100 
Student READ 180 1.95 1.14 374 1.70 0.089 
Gender 1.52 1.10 374 1.38 0.169 
ELL 9.90 4.49 374 2.21 0.028 
African American 11.54 4.72 374 2.45 0.015 
ITBS Comprehension Pre 0.36 0.05 374 7.04 0.000 
Random Effects 
Level Variance Components Variance 
School School intercepts 0.03 
Student Residual 114.75 
Random Effects (From Unconditional Model) 
Level Variance Components Variance icc! 
School School intercepts 0.03 0.000 
Student Residual 133.86 


* Final model includes the treatment effect of READ 180 and all covariates with p<0.2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 
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Table G-45: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on ITBS Vocabulary Scores of Sixth-Graders in Year 4 


Fixed Effects 
Level Effect Estimate St Err DF | t-value Pr > |t| 
School Intercept 21.06 0.76 3 27.56 0.000 
Percent Female 0.97 0.48 3 2.03 0.127 
Percent SPED 0.82 0.44 3 1.87 0.153 
Percent ELL 1.04 0.32 3 3.29 0.071 
Percent FRL -0.94 0.49 3 -1.92 0.145 
Student READ 180 -0.58 1.05 374 -0.55 0.583 
Gender -2.52 1.02 374 -2.48 0.014 
African American 7.27 2.40 374 3.03 0.003 
ITBS Vocabulary Pre 0.30 0.04 374 7.08 0.000 
Random Effects 
Level Variance Components Variance 
School School intercepts 0.02 
Student Residual 96.38 
Random Effects (From Unconditional Model) 
Level Variance Components Variance icc’ 
School School intercepts 3.32 0.028 
Student Residual 117.08 


* Final model includes the treatment effect of READ 180 and all covariates with p<0. 2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 


Table G-46: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on ITBS Total Reading Scores of Sixth-Graders in Years 1-4 


Fixed Effects 
Level Effect Estimate St Err DF t-value | Pr > |t| 

School Intercept 19.37 0.33 5 59.14 0.000 
Percent Female 0.52 0.19 5 2.81 0.038 
Percent ELL 0.36 0.09 5 3.93 0.016 

Student READ 180 -0.29 0.46 1710 -0.63 0.530 
ELL -2.73 1.04 1710 -2.63 0.009 
ITBS Total Reading Pre 0.44 0.02 1710 20.17 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 0.02 

Student Residual 87.83 

Random Effects (From Unconditional Model) 

Level Variance Components Variance icc! 

School School intercepts 0.64 0.006 

Student Residual 109.87 


* Final model includes the treatment effect of READ 180 and all covariates with p<0.2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 
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Table G-47: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on ITBS Comprehension Scores of Sixth-Graders in Years 1-4 


Fixed Effects 


Level Effect Estimate St Err DF t-value | Pr> |t| 

School Intercept 22.13 0.37 5 | 60.00 0.000 
Percent Female 0.28 0.16 5 1.69 0.152 
Size 0.00 0.00 5 2.10 0.088 

Student READ 180 -0.12 0.51 1750 -0.24 0.814 
Gender 1.77 0.49 1750 3.61 0.001 
ITBS Comprehension Pre 0.37 0.02 1750 | 16.12 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 0.02 

Student Residual 105.24 

Random Effects (From Unconditional Model) 

Level Variance Components Variance icc’ 

School School intercepts 0.29 0.002 

Student Residual 120.81 


* Final model includes the treatment effect of READ 180 and all covariates with p<0. 2. 


” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 


Table G-48: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on ITBS Vocabulary Scores of Sixth-Graders in Years 1-4 


Fixed Effects 


Level Effect Estimate St Err DF t-value | Pr > |t| 

School Intercept 21.00 0.35 5 60.57 0.000 
Percent Female 0.56 0.19 5 2.90 0.035 
Percent ELL 0.47 0.10 5 4.90 0.003 

State READ 180 0.02 0.50 1726 0.03 0.975 
Gender -1.92 0.49 1726 -3.95 0.000 
ELL -3.44 1.10 1726 -3.11 0.002 
ITBS Vocabulary Pre 0.35 0.02 1726 16.85 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 0.01 

Student Residual 101.02 

Random Effects (From Unconditional Model) 

Level Variance Components Variance icc’ 

School School intercepts 1.29 0.011 

Student Residual 120.20 


* Final model includes the treatment effect of READ 180 and all covariates with p<0.2. 


” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 


Appendix G. MSRP Evaluation Report, Year 4 


March 1, 2011. Page 26 of 44 


Table G-49: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on ITBS Total Reading Scores of Sixth-Graders in Years 1—4 and Seventh- and 
Eighth-Graders in Year 1 


Fixed Effects 


Level Effect Estimate St Err DF t-value | Pr > |t| 

School Intercept 20.11 0.36 5 55.23 0.000 
Percent Female 0.49 0.25 5 1.95 0.108 
Percent ELL 0.33 0.12 5 2.82 0.038 

State READ 180 -0.40 0.41 2467 -0.99 0.322 
FRL -0.98 0.73 2467 -1.35 0.177 
ELL -2.02 0.87 2467 -2.32 0.021 
ITBS Total Reading Pre 0.41 0.02 2467 22.32 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 0.41 

Student Residual 95.02 

Random Effects (From Unconditional Model) 

Level Variance Components Variance icc! 

School School intercepts 1.02 0.009 

Student Residual 114.65 


* Final model includes the treatment effect of READ 180 and all covariates with p<0. 2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 


Table G-50: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on ITBS Comprehension Scores of Sixth-Graders in Years 1—4 and Seventh- and 
Eighth-Graders in Year 1 


Fixed Effects 


Level Effect Estimate St Err DF t-value | Pr> |t| 

School Intercept 22.62 0.41 6 55.06 0.000 
Size 0.00 0.00 6 1.51 0.181 

State READ 180 -0.12 0.44 2518 -0.27 0.790 
Gender 1.78 0.42 2518 4.27 0.000 
FRL -1.01 0.77 2518 -1.30 0.193 
ITBS Comprehension Pre 0.34 0.02 2518 18.02 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 0.59 

Student Residual 109.00 

Random Effects (From Unconditional Model) 

Level Variance Components Variance icc’ 

School School intercepts 0.68 0.006 

Student Residual 122.32 


* Final model includes the treatment effect of READ 180 and all covariates with p<0. 2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 
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Table G-51: Final Model* for Estimating Fixed and Random Effects for Determining the One-Year 
Impact of READ 180 on ITBS Vocabulary Scores of Sixth-Graders in Years 1—4 and Seventh- and 
Eighth-Graders in Year 1 


Fixed Effects 


Level Effect Estimate St Err DF t-value | Pr > |t| 

School Intercept 21.84 0.30 5 71.96 0.000 
Percent Female 0.39 0.17 5 2.25 0.073 
Percent ELL 0.39 0.09 5 4.54 0.006 

State READ 180 -0.24 0.44 2501 -0.54 0.590 
Gender -1.81 0.43 2501 -4.16 0.000 
African American 2.63 0.94 2501 2.79 0.006 
ITBS Vocabulary Pre 0.33 0.02 2501 18.89 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 0.00 

Student Residual 116.71 

Random Effects (From Unconditional Model) 

Level Variance Components Variance icc! 

School School intercepts 1.25 0.009 

Student Residual 134.47 


* Final model includes the treatment effect of READ 180 and all covariates with p<0.2. 


” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 


Table F-52: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 ITBS Total Reading Scores of Seventh- and Eighth-Graders in Year 2 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 198.05 0.56 5 221.71 0.000 
Percent Female 0.76 0.45 5 1.70 0.149 
Percent FRL -0.76 0.32 5 -2.37 0.062 

Student | Grade 8 in Year 2 4.23 1.26 652 3.35 0.001 
READ 180 0.05 1.28 652 0.04 0.967 
ITBS Total Reading 06 0.49 0.04 652 11.66 0.000 

Random Effects 

Level Variance Components Variance icc’ 

School Level 2 1.03 0.016 

Student | Level 1 247.36 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-53: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 ITBS Comprehension Scores of Seventh- and Eighth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 193.66 1.00 5 192.81 0.000 
Percent African American -0.44 0.21 5 -2.15 0.083 
Percent Female 2.01 0.59 5 3.41 0.024 

Student | Grade 8 in Year 2 2.83 1.54 658 1.84 0.065 
READ 180 -0.74 1.58 658 -0.47 0.639 
Gender 2.89 1.53 658 1.89 0.059 
ITBS Comprehension 06 0.39 0.04 658 8.62 0.000 

Random Effects 

Level Variance Components Variance Icct 

School Level 2 0.08 0.014 

Student | Level 1 385.10 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-54: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 ITBS Vocabulary Scores of Seventh- and Eighth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 201.47 1.05 6 190.28 0.000 
Percent FRL -1.18 0.37 6 -3.22 0.021 
Student | Grade 8 in Year 2 7.47 1.66 666 4.49 0.000 
READ 180 1.79 1.65 666 1.08 0.280 
ELL -8.72 6.01 666 -1.45 0.147 
ITBS Vocabulary 06 0.31 0.04 666 7.40 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.25 0.012 
Student | Level 1 428.05 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-55: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Reading/LA Scores of Seventh- and Eighth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 505.29 1.15 3 437.57 0.000 
Percent African American -1.33 0.34 3 -3.92 0.059 
Percent Female 2.17 0.71 3 3.05 0.068 
Percent SPED 2.28 0.73 3 3.11 0.069 
Percent FRL -2.66 0.81 3 -3.29 0.071 
Student | Grade 8 in Year 2 16.48 1.76 804 9.35 0.000 
READ 180 1.38 1.81 804 0.76 0.446 
Gender 3.51 1.76 804 2.00 0.045 
ELL -9.94 3.58 804 -2.78 0.006 
TCAP Reading 06 0.25 0.04 804 7.02 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.10 0.021 
Student | Level 1 617.56 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-56: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Mathematics Scores of Seventh- and Eighth-Graders in Year 2 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 511.88 3.02 7 169.52 0.000 
Student | Grade 8 in Year 2 4.43 2.09 808 2.12 0.034 

READ 180 -0.35 2.16 808 -0.16 0.871 
Gender 4.26 2.08 808 2.04 0.041 
TCAP Math 06 0.57 0.04 808 15.60 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 57.32 0.060 
Student | Level 1 861.99 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-57: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Science Scores of Seventh- and Eighth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 184.39 0.74 4 250.68 0.000 
Percent Female 1.63 0.46 4 3.53 0.036 
Percent SPED 0.37 0.23 4 1.58 0.188 
Percent ELL 0.44 0.27 4 1.62 0.179 
Student | Grade 8 in Year 2 -2.82 1.12 780 -2.52 0.012 
READ 180 0.32 1.14 780 0.28 0.782 
Gender -1.73 1.11 780 -1.55 0.121 
African American -6.39 4.06 780 1.58 0.115 
ELL -6.37 4.24 780 -1.50 0.133 
TCAP Science 06 0.32 0.03 780 9.17 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.06 0.027 
Student | Level 1 240.01 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-58: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Social Studies Scores of Seventh- and Eighth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 186.53 0.60 3 310.21 0.000 
Percent African American -1.27 0.57 3 -2.22 0.103 
Percent ELL -1.76 0.77 3 -2.27 0.097 
Percent FRL -0.79 0.20 3 -4.01 0.054 
Enrollment -0.01 0.003 3 -2.62 0.072 
Student | READ 180 -1.37 0.90 776 -1.52 0.129 
African American -4.59 1.90 776 -2.41 0.016 
TCAP Social Studies 06 0.14 0.02 776 5.82 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.07 0.035 
Student | Level 1 145.89 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-59: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 ITBS Total Reading Scores of Seventh-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 193.29 1.03 6 187.70 0.000 
Percent FRL -0.83 0.36 6 -2.29 0.061 
Student | READ 180 1.99 1.64 330 1.22 0.226 
Gender 3.08 1.59 330 1.94 0.053 
African American -8.95 6.06 330 1.94 0.053 
ELL -14.38 6.52 330 -2.21 0.028 
ITBS Total Reading 06 0.57 0.06 330 9.29 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.10 0.006 
Student | Level 1 205.86 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-60: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 ITBS Comprehension Scores of Seventh-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 190.50 1.30 4 146.71 0.000 
Percent SPED 1.36 0.78 4 1.74 0.155 
Percent ELL 0.85 0.47 4 1.83 0.140 
Percent FRL -2.01 0.84 4 -2.39 0.072 
Student | READ 180 -0.03 2.05 329 -0.02 0.988 
Gender 4.02 1.98 329 2.04 0.042 
African American -36.67 18.01 329 -2.04 0.042 
Hispanic -28.03 19.72 329 -1.42 0.156 
FRL -5.95 3.09 329 -1.93 0.055 
ELL -13.01 8.65 329 1.50 0.133 
ITBS Comprehension 06 0.51 0.07 329 7.81 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.15 0.010 
Student | Level 1 318.09 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-61: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 ITBS Vocabulary Scores of Seventh-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 195.20 1.37 6 142.15 0.000 
Percent FRL -0.94 0.48 6 -1.93 0.101 
Student | READ 180 4.39 2.19 339 2.00 0.045 
Hispanic 12.52 9.02 339 1.39 0.166 
ELL -20.67 9.37 339 -2.21 0.028 
ITBS Vocabulary 06 0.35 0.06 339 5.82 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.08 0.007 
Student | Level 1 380.32 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-62: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Reading/LA Scores of Seventh-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 497.21 1.80 6 276.54 0.000 
Percent FRL -1.48 0.63 6 -2.35 0.056 
Student | READ 180 1.39 2.87 409 0.48 0.629 
Gender 6.11 2.79 409 2.19 0.029 
African American -14.32 10.83 409 -1.32 0.187 
ELL -28.22 11.63 409 -2.43 0.016 
TCAP Reading 06 0.28 0.06 409 4.56 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.50 0.012 
Student | Level 1 787.39 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-63: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Mathematics Scores of Seventh-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 507.80 5.09 7 99.76 0.000 
Student | READ 180 -0.80 2.98 412 -0.27 0.788 
Gender 4.38 2.87 412 1.53 0.128 
TCAP Math 06 0.64 0.05 412 11.69 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 177.62 0.137 
Student | Level 1 824.04 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-64: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Science Scores of Seventh-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 186.95 1.05 5 178.08 0.000 
Percent SPED 1.29 0.41 5 3.13 0.030 
Percent FRL -1.43 0.55 5 -2.63 0.046 

Student | READ 180 -0.72 1.54 396 -0.47 0.639 
Gender -2.71 1.50 396 -1.80 0.072 
African American -9.40 5.87 396 -1.60 0.109 
FRL -3.75 2.46 396 -1.52 0.128 
ELL -9.88 6.43 396 -1.54 0.125 
TCAP Science 06 0.30 0.05 396 5.72 0.000 

Random Effects 

Level Variance Components Variance icc’ 

School Level 2 0.99 0.039 

Student | Level 1 221.21 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-65: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Social Studies Scores of Seventh-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 186.51 1.13 5 165.63 0.000 
Percent ELL -0.49 0.31 5 -1.61 0.169 
Percent FRL -0.82 0.43 5 -1.88 0.118 
Student | READ 180 -1.10 1.20 399 -0.92 0.361 
African American -5.93 2.97 399 -1.99 0.046 
TCAP Social Studies 06 0.14 0.03 399 4.421 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 5.11 0.054 
Student | Level 1 134.74 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-66: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 ITBS Total Reading Scores of Eighth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 202.88 1.78 7 114.79 0.000 
Student | READ 180 -2.20 1.95 317 -1.13 0.260 
Hispanic 5.59 3.52 317 1.59 0.114 
ITBS Total Reading 06 0.43 0.06 317 7.38 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School | Level 2 12.57 0.044 
Student | Level 1 278.72 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-67: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 ITBS Comprehension Scores of Eighth-Graders in Year 2 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 197.13 1.51 6 130.66 0.000 
Percent Female 1.97 0.72 6 2.73 0.035 
Student | READ 180 -2.73 2.39 321 -1.14 0.256 
ITBS Comprehension 06 0.32 0.06 321 5.18 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.05 0.012 
Student | Level 1 442.60 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-68: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 ITBS Vocabulary Scores of Eighth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 208.07 2.64 7 78.89 0.000 
Student | READ 180 -1.44 2.44 324 -0.59 0.555 
African American -13.18 4.43 324 -2.98 0.004 
ITBS Vocabulary 06 0.28 0.05 324 4.97 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 35.92 0.061 
Student | Level 1 444.24 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-69: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Reading/LA Scores of Eighth-Graders in Year 2 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 514.23 1.43 3 360.57 0.000 
Percent African American -1.34 0.42 3 -3.18 0.069 
Percent Female 2.52 0.90 3 2.80 0.067 
Percent SPED 2.62 0.90 3 2.93 0.067 
Percent FRL -2.61 0.99 3 -2.64 0.071 
Student | READ 180 1.17 2.18 389 0.537 0.591 
FRL 4.18 3.24 389 1.288 0.199 
ELL -5.84 3.98 389 -1.466 0.143 
TCAP Reading 06 0.23 0.04 389 5.656 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School Level 2 0.60 0.052 
Student | Level 1 443.87 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-70: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Mathematics Scores of Eighth-Graders in Year 2 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 
School Intercept 516.79 2.42 7 213.31 0.000 
Student | READ 180 -0.13 3.03 391 -0.04 0.966 
Gender 5.07 2.94 391 1.72 0.085 
African American 49.51 29.19 391 1.70 0.090 
Hispanic 45.47 29.53 391 1.54 0.124 
TCAP Math 06 0.52 0.05 391 11.05 0.000 
Random Effects 

Level Variance Components Variance icc’ 

School | Level 2 16.95 0.020 

Student | Level 1 840.77 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-71: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Science Scores of Eighth-Graders in Year 2 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 181.81 1.36 5 134.01 0.000 
Percent African American -0.71 0.31 5 -2.30 0.068 
Percent Female 2.77 0.94 5 2.96 0.034 

Student | READ 180 1.62 1.66 380 0.98 0.331 
TCAP Science 06 0.33 0.05 380 7.32 0.000 

Random Effects 

Level Variance Components Variance icc’ 

School Level 2 5.57 0.062 

Student | Level 1 246.28 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-72: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on 2008 TCAP Social Studies Scores of Eighth-Graders in Year 2 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 186.60 0.89 3 210.37 0.000 
Percent African American -0.51 0.24 3 -2.10 0.118 
Percent Female 1.00 0.54 3 1.85 0.156 
Percent FRL -0.78 0.30 3 -2.64 0.071 
Enrollment -0.01 0.004 3 -1.88 0.151 

Student | READ 180 -1.52 1.35 372 -1.13 0.259 
TCAP Social Studies 06 0.14 0.03 372 4.23 0.000 

Random Effects 

Level Variance Components Variance icc’ 

School Level 2 0.08 0.029 

Student | Level 1 157.04 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Impact of READ 180 on Spring 2009 ITBS Reading Total Scores of Seventh-Graders in Year 3 


Fixed Effects 
Level Effect Estimate St Err DF t-value Pr > |t| 
School Intercept 194.87 1.05 6 185.57 0.000 
Percent African American -0.53 0.16 6 -3.32 0.019 
Student READ 180 -0.18 1.61 346 -0.11 0.910 
Gender 2.54 1.55 346 1.64 0.103 
Free/Reduced Lunch 6.35 3.52 346 1.80 0.072 
ITBS Reading Total 07 0.56 0.06 346 8.88 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School School intercepts 0.04 0.017 
Student Residual 208.53 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 


Table G-74: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 


Impact of READ 180 on Spring 2009 ITBS Comprehension Scores of Seventh-Graders in Year 3 


Fixed Effects 
Level Effect Estimate St Err DF t-value Pr > |t| 

School Intercept 190.98 1.37 6 139.01 0.000 
Percent English Language 
Learners 0.59 0.30 6 1.99 0.093 

Student READ 180 -2.35 2.11 347 -1.12 0.266 
Gender 4.39 2.04 347 2.16 0.032 
Free/Reduced Lunch 7.37 4.65 347 1.59 0.113 
ITBS Comprehension 07 0.51 0.07 347 7.62 0.000 

Random Effects 

Level Variance Components Variance icc’ 

School School intercepts 0.05 0.026 

Student Residual 362.75 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-75: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on Spring 2009 ITBS Vocab Scores of Seventh-Graders in Year 3 


Fixed Effects 
Level Effect Estimate St Err DF t-value Pr > |t| 
School Intercept 198.57 1.20 5 164.87 0.000 
Percent African American -3.17 1.05 5 -3.03 0.032 
Percent English Language 
Learners -3.72 1.46 5 -2.54 0.051 
Student READ 180 1.27 1.83 370 0.70 0.487 
Free/Reduced Lunch 6.62 4.01 370 1.65 0.099 
ITBS Vocab 07 0.36 0.05 370 6.63 0.000 
Random Effects 
Level Variance Components Variance icc! 
School School intercepts 0.04 0.026 
Student Residual 286.76 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-76: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on Spring 2009 TCAP Reading Scores of Seventh-Graders in Year 3 


Fixed Effects 
Level Effect Estimate St Error DF t-value Pr > |t| 
School Intercept 496.96 2.02 7 246.05 0.000 
Student READ 180 -1.19 2.47 571 -0.48 0.629 
Gender 3.90 2.40 571 1.63 0.104 
TCAP Reading 07 0.47 0.05 571 9.16 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School School intercepts 11.65 0.008 
Student Residual 806.63 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Table G-77: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on Spring 2009 TCAP Math Scores of Seventh-Graders in Year 3 


Fixed Effects 
Level Effect Estimate St Err DF t-value Pr > |t| 

School Intercept 508.17 3.48 6 146.14 0.000 
Percent Free/Reduced 
Lunch -2.23 1.41 6 -1.59 0.163 

Student READ 180 -2.66 2.46 568 -1.08 0.281 
Gender 3.10 2.35 568 1.32 0.189 
Free/Reduced Lunch 8.52 5.06 568 1.68 0.092 
English Language Learner -8.60 6.11 568 -1.41 0.160 
TCAP Math 07 0.63 0.05 568 13.06 0.000 

Random Effects 

Level Variance Components Variance icc! 

School School intercepts 74.66 0.052 

Student Residual 783.80 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 
” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 


Table G-78: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on Spring 2009 TCAP Science Scores of Seventh-Graders in Year 3 


Fixed Effects 


Level Effect Estimate St Error DF t-value Pr > |t| 
School Intercept 185.18 1.59 7 116.23 0.000 
Student READ 180 -2.21 1.53 562 -1.44 0.149 

English Language Learner -10.50 7.32 562 -1.44 0.152 
African American -11.24 6.77 562 -1.66 0.097 
TCAP Science 07 0.32 0.04 562 7.13 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School School intercepts 11.95 0.022 
Student Residual 302.99 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 
final model results displayed in the table. 
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Impact of READ 180 on Spring 2009 TCAP Social Studies Scores of Seventh-Graders in Year 3 


Fixed Effects 


Level Effect Estimate St Error DF t-value Pr > |t| 
School Intercept 186.81 1.67 7 111.54 0.000 
Student READ 180 -1.26 0.99 559 -1.28 0.202 

Gender -1.88 0.94 559 -1.99 0.047 
Free/Reduced Lunch 3.25 2.03 559 1.60 0.109 
TCAP Social Studies 07 0.42 0.04 559 11.44 0.000 
Random Effects 
Level Variance Components Variance icc’ 
School School intercepts 18.92 0.094 
Student Residual 122.65 


* Final model includes the treatment effect of READ 180 and all covariates with p < 0.2. 


” The interclass correlation coefficient was calculated using a model with no covariates. The variance components are based on the 


final model results displayed in the table. 


Table G-80: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 


Impact of READ 180 on ITBS Total Reading Scores of Seventh-Graders in Year 4 


Fixed Effects 


Level Effect Estimate St Err DF | t-value | Pr>|t| 

School Intercept 20.35 1.11 3 18.33 0.000 
Percent African American 2.96 0.77 3 3.86 0.061 
Percent Gender 1.72 0.53 3 3.22 0.070 
Percent ELL 4.13 1.12 3 3.70 0.067 
Size 0.01 0.00 3 2.43 0.083 

Student READ 180 1.20 1.32 254 0.91 0.364 
Gender 2.10 1.27 254 1.65 0.100 
African American -5.25 2.75 254 -1.91 0.057 
ITBS Total Reading Pre 0.51 0.07 254 7.59 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 0.04 

Student Residual 104.12 

Random Effects (From Unconditional Model) 

Level Variance Components Variance icct 

School School intercepts 3.78 0.029 

Student Residual 127.14 


* Final model includes the treatment effect of READ 180 and all covariates with p<0.2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 


Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 
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Table G-81: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on ITBS Comprehension Scores of Seventh-Graders in Year 4 


Fixed Effects 


Level Effect Estimate StError | DF | t-value | Pr> |t| 
School Intercept 22.97 1.26 7 | 18.29 0.000 
Student READ 180 2.39 1.29 271 1.86 0.064 
Gender 1.98 1.27 271 1.56 0.120 
FRL -3.62 2.52 271 -1.43 0.153 
African American -4.12 2.66 271 -1.55 0.123 
ITBS Comprehension Pre 0.32 0.06 271 5.17 0.000 
Random Effects 

Level Variance Components Variance 

School School intercepts 4.56 

Student Residual 106.95 

Random Effects (From Unconditional Model) 

Level Variance Components Variance icc’ 

School School intercepts 3.66 0.030 

Student Residual 118.52 


* Final model includes the treatment effect of READ 180 and all covariates with p<0. 2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 


Table G-82: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on ITBS Vocabulary Scores of Seventh-Graders in Year 4 


Fixed Effects 


Level Effect Estimate St Err DF | t-value | Pr>|t| 

School Intercept 22.13 1.41 4 | 15.71 0.000 
Percent African American 2.19 1.18 4 1.86 0.135 
Percent Female 1.83 0.82 4 2.24 0.085 
Percent ELL 3.68 1.75 4 2.10 0.101 

Student READ 180 0.00 1.59 259 0.00 0.999 
ITBS Vocabulary Pre 0.44 0.07 259 6.11 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 2.45 

Student Residual 155.43 

Random Effects (From Unconditional Model) 

Level Variance Components Variance icc’ 

School School intercepts 1.33 0.007 

Student Residual 177.04 


* Final model includes the treatment effect of READ 180 and all covariates with p<0.2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 
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Table G-83: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on ITBS Comprehension Scores of Seventh- and Eighth-Graders in Year 2 and 
Seventh-Graders in Years 3 and 4 


Fixed Effects 


Level Effect Estimate St Err DF t-value | Pr > |t| 

School Intercept 24.53 0.43 5 | 56.84 0.000 
Percent Female 0.88 0.25 5 3.44 0.024 
Percent ELL 0.36 0.12 5 2.90 0.035 

Student READ 180 -0.10 0.63 1289 | -0.15 0.878 
Gender 2.00 0.62 1289 3.25 0.002 
ITBS Comprehension Pre 0.32 0.03 1289 | 11.36 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 0.05 

Student Residual 122.37 

Random Effects (From Unconditional Model) 

Level Variance Components Variance Icc** 

School School intercepts 1.22 0.009 

Student Residual 135.26 


* Final model includes the treatment effect of READ 180 and all covariates with p<0. 2. 


” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 


Table G-84: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on ITBS Total Reading Scores of Seventh- and Eighth-Graders in Year 2 and 
Seventh-Graders in Years 3 and 4 


Fixed Effects 


Level Effect Estimate St Err DF t-value | Pr> |t| 

School Intercept 21.75 0.43 5 50.35 0.000 
Percent Female 0.91 0.26 5 3.50 0.023 
Percent ELL 0.42 0.13 5 3.30 0.026 

Student READ 180 0.18 0.61 1267 0.29 0.769 
Gender 1.65 0.60 1267 2.76 0.006 
ITBS Total Reading Pre 0.43 0.03 1267 15.32 0.000 

Random Effects 

Level Variance Components Variance 

School School intercepts 0.13 

Student Residual 113.07 

Random Effects (From Unconditional Model) 

Level Variance Components Variance Icc** 

School School intercepts 1.20 0.009 

Student Residual 133.25 


* Final model includes the treatment effect of READ 180 and all covariates with p<0. 2. 


” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 
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Table G-85: Final Model* for Estimating Fixed and Random Effects for Determining the Two-Year 
Impact of READ 180 on ITBS Vocabulary Scores of Seventh- and Eighth-Graders in Year 2 and 
Seventh-Graders in Years 3 and 4 


Fixed Effects 
Level Effect Estimate St Err DF t-value | Pr> |t| 
School Intercept 22.47 0.46 6 48.84 0.000 
FRL -0.44 0.16 6 -2.80 0.032 
State READ 180 0.73 0.69 1309 1.05 0.294 
African American -3.17 1.44 1309 -2.20 0.028 
ITBS Vocabulary Pre 0.32 0.03 1309 11.44 | 0.000 
Random Effects 
Level Variance Components Variance 
School School intercepts 0.02 
Student Residual 153.61 
Random Effects (From Unconditional Model) 
Level Variance Components Variance Icc** 
School School intercepts 0.99 0.006 
Student Residual 168.70 


* Final model includes the treatment effect of READ 180 and all covariates with p<0. 2. 
” The interclass correlation coefficient was calculated using a (unconditional) model with no covariates. 
Note: Due to missing pretest data in the final model, the sample sizes for the final and unconditional models differ. 
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Appendix H 
Tests of Differential Attrition in the READ 180 Impact ITT Student Samples 
Table H-1: Tests of Differential Attrition in the READ 180 Impact Student Sample 
Nee | emer oe | Amweieteae || eee |) geet 
Dependent Variable Differential Favored by 
Treatment Control Treatment Control Treatment Control Attrition Differential 
Group Group Group Group Group Group Effect ‘ Attrition 
; 145 228 
ITBS Total Reading 656 940 (22%) (24%) 0.1 -0.3 0.429 Treatment 
TCAP Reading/LA 698 1042 i ee 0.2 -0.1 0.211 Treatment 
(5%) (7%) 
TCAP Mathematics 697 1040 a oF 0.6 0.4 0.369 Treatment 
(5%) (7%) 
TCAP Science 686 1006 ais Y -0.1 0.3 0.164 Control 
(6%) (9%) 
TCAP Social Studies 685 1007 7 ie 0.1 0.6 0.195 Control 
(6%) (10%) 


*Attrition Effect = Mean pretest score of students with a posttest score minus mean pretest score of all students (how much higher/lower the mean pretest score 


was as a result of the attrition) 
Differential Attrition Effect = Difference between treatment and control group attrition effects 
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Appendix I: Complete Results of Multi-Level Analyses of 
READ 180 Impact on Student Achievement in MCLA Treatment 
and Control Schools in Year 1 


Table I-1 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS Total Reading 
Scores—Year 1, All Grades 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 191.33 1.01 3 189.99 0.000 
MCLA 2.03 1.49 3 1.36 0.267 
Percent African American 1.23 0.61 3 2.01 0.130 
Percent ELL 2.28 0.95 3 2.39 0.086 
Percent FRL -0.49 0.20 3 -2.45 0.081 

Student | Grade 7 5.66 0.94 1212 6.01 0.000 
Grade 8 12.32 1.07 1212 11.49 0.000 
READ 180 -0.75 1.13 1212 -0.67 0.505 
READ 180*MCLA 1.04 1.62 1212 0.65 0.518 
ELL -3.65 1.66 1212 -2.19 0.028 
ITBS Total Reading 06 0.43 0.03 1212 16.03 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Intercept 0.04 0.005 

Student | Level 1 185.30 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-2 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS 
Comprehension Scores—Year 1, All Grades 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 185.75 1.26 3 147.25 0.000 
MCLA 2.66 1.87 3 1.42 0.249 
Percent African American 1.83 0.76 3 2.39 0.086 
Percent ELL 3.07 1.19 3 2.59 0.073 
Percent FRL -1.01 0.25 3 -4.12 0.048 

Student | Grade 7 7.11 1.15 1222 6.16 0.000 
Grade 8 11.11 1.28 1222 8.71 0.000 
READ 180 -1.46 1.41 1222 -1.03 0.303 
READ 180*MCLA 2.79 2.03 1222 1.38 0.168 
Gender 3.78 0.99 1222 3.83 0.000 
Hispanic 5.53 3.80 1222 1.45 0.146 
ELL -7.97 3.86 1222 -2.07 0.039 
ITBS Comprehension 06 0.37 0.03 1222 12.96 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 0.04 0.008 

Student | Level 1 293.89 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-3 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS Vocabulary 
Scores—Year 1, All Grades 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 197.25 1.12 5 175.86 0.000 
MCLA 0.40 1.54 5 0.26 0.808 
Percent ELL 0.50 0.19 5 2.59 0.048 

Student | Grade 7 5.26 1.22 1235 4.32 0.000 
Grade 8 15.96 1.36 1235 11.73 0.000 
READ 180 0.42 1.44 1235 0.29 0.773 
READ 180*MCLA -0.65 2.05 1235 -0.32 0.753 
Gender -2.13 1.01 1235 -2.10 0.036 
Hispanic -4.70 2.10 1235 -2.24 0.025 
ITBS Vocabulary 06 0.35 0.03 1235 12.97 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 0.02 0.003 

Student | Level 1 310.83 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 
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Table I-4 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Reading/LA 
Scores—Year 1, All Grades 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 499.02 1.49 3 335.34 0.000 
MCLA -4.16 2.17 3 -1.92 0.145 
Percent African American 2.26 1.00 3 2.25 0.100 
Percent Female 2.78 0.54 3 5.16 0.006 
Percent ELL 3.85 1.59 3 2.42 0.083 

Student | Grade 7 8.77 1.51 1622 5.81 0.000 
Grade 8 13.59 1.62 1622 8.38 0.000 
READ 180 -1.37 1.87 1622 -0.73 0.464 
READ 180*MCLA 3.09 2.65 1622 1.17 0.245 
Gender 4.00 1.29 1622 3.12 0.002 
Hispanic 8.49 4.94 1622 1.72 0.085 
FRL -5.15 2.05 1622 -2.52 0.012 
ELL -15.30 5.08 1622 3.01 0.003 
TCAP Reading 06 0.33 0.03 1622 12.84 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Intercept 0.07 0.012 

Student | Level 1 657.62 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-5 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Mathematics 
Scores—Year 1, All Grades 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 501.94 1.75 6 287.04 0.000 
MCLA -3.18 2.44 6 -1.30 0.241 

Student | Grade 7 2.70 1.62 1621 1.66 0.096 
Grade 8 7.91 1.72 1621 4.60 0.000 
READ 180 2.48 2.00 1621 1.24 0.215 
READ 180*MCLA -5.37 2.83 1621 -1.90 0.058 
Gender 5.39 1.38 1621 3.92 0.000 
African American 67.41 19.49 1621 3.46 0.001 
Hispanic 77.83 20.14 1621 3.86 0.000 
ELL -7.32 5.44 1621 -1.35 0.179 
TCAP Math 06 0.51 0.02 1621 24.13 0.000 

Random Effects 

Level Variance Components Variance icc * 

School Intercept 4.90 0.021 

Student | Level 1 753.57 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-6 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Science 
Scores—Year 1, All Grades 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 185.85 0.84 5 221.43 0.000 
MCLA -0.38 1.11 5 -0.34 0.747 
Size -0.01 0.00 5 -2.96 0.033 

Student | Grade 7 -2.41 0.86 1550 -2.80 0.006 
READ 180 0.19 1.20 1550 0.16 0.874 
READ 180*MCLA -1.44 1.67 1550 -0.86 0.388 
ELL -4.86 2.48 1550 -1.96 0.050 
TCAP Science 06 0.22 0.02 1550 9.39 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 0.12 0.008 

Student | Level 1 258.84 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Table I-7 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Social Studies 
Scores—Year 1, All Grades 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 185.69 1.72 6 107.93 0.000 
MCLA -1.32 2.41 6 -0.55 0.604 

Student | READ 180 0.81 1.19 1544 0.69 0.493 
READ 180*MCLA -0.00 1.67 1544 -0.00 0.998 
African American 3.09 2.17 1544 1.43 0.154 
TCAP Social Studies 06 0.22 0.02 1544 9.39 0.000 

Random Effects 

Level Variance Components Variance icc t 

School Intercept 9.10 0.034 

Student | Level 1 246.36 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-8 Final Model* for Estimating Fixed and Random Effects for Determining 
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READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS Total Reading 
Scores—Year 1, Sixth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 183.79 0.94 6 194.67 0.000 
MCLA -0.03 1.33 6 -0.03 0.980 

Student | READ 180 -0.13 1.45 458 -0.09 0.928 
READ 180*MCLA 1.33 2.11 458 0.63 0.529 
FRL -3.02 1.65 458 -1.83 0.067 
ELL -5.54 2.19 458 -2.53 0.012 
ITBS Total Reading 06 0.53 0.04 458 13.07 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 0.03 0.000 

Student | Level 1 119.68 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Table I-9 Final Model* for Estimating Fixed and Random Effects for Determining 


READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS 
Comprehension Scores—Year 1, Sixth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 179.36 1.20 6 149.62 0.000 
MCLA 1.86 1.69 6 1.10 0.314 

Student | READ 180 -0.96 1.85 458 -0.52 0.605 
READ 180*MCLA 0.18 2.68 458 0.07 0.947 
Gender 4.23 1.31 458 3.23 0.002 
African American -10.80 5.80 458 -1.86 0.063 
FRL -3.91 2.10 458 -1.87 0.062 
ELL -15.03 6.00 458 -2.51 0.013 
ITBS Comprehension 06 0.41 0.04 458 9.36 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 0.02 0.003 

Student | Level 1 192.95 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
" The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 
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Table I-10 Final Model* for Estimating Fixed and Random Effects for Determining 
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READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS Vocabulary 
Scores—Year 1, Sixth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 186.73 1.49 5 125.74 0.000 
MCLA -0.03 2.08 5 -0.02 0.988 
Percent ELL 0.58 0.25 5 2.26 0.071 

Student | READ 180 1.31 1.95 462 0.67 0.501 
READ 180*MCLA 3.18 2.81 462 1.13 0.258 
Gender -2.40 1.38 462 -1.74 0.082 
ELL -7.35 2.90 462 -2.53 0.012 
ITBS Vocabulary 06 0.39 0.04 462 9.84 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 0.02 0.012 

Student | Level 1 212.91 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-11 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Reading/LA 


Scores—Year 1, Sixth Grade 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 
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Table I-12 Final Model* for Estimating Fixed and Random Effects for Determining 
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READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Mathematics 
Scores—Year 1, Sixth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 490.07 2.50 5 196.12 0.000 
MCLA 4.27 3.49 5 1.22 0.275 
Size 0.02 0.01 5 2.63 0.046 

Student | READ 180 5.31 3.07 592 1.73 0.084 
READ 180*MCLA -6.03 4.35 592 -1.39 0.166 
Gender 4.42 2.10 592 2.10 0.036 
TCAP Math 06 0.53 0.04 592 13.41 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 9.11 0.018 

Student | Level 1 641.60 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Table I-13 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Science 


Scores—Year 1, Sixth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 183.13 1.82 3 100.66 0.000 
MCLA 6.76 2.97 3 2.28 0.096 
Percent Female -1.65 0.81 3 -2.04 0.126 
Percent FRL -0.91 0.49 3 -1.85 0.156 
Size -0.01 0.004 3 -2.98 0.067 

Student | READ 180 3.78 1.86 564 2.04 0.042 
READ 180*MCLA -5.41 2.58 564 -2.09 0.037 
Gender -1.84 1.27 564 -1.45 0.148 
TCAP Science 06 0.27 0.04 564 6.73 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 0.08 0.012 

Student | Level 1 223.68 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 
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Table I-14 Final Model* for Estimating Fixed and Random Effects for Determining 
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READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Social Studies 
Scores—Year 1, Sixth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 184.98 2.34 6 79.18 0.000 
MCLA 2.61 3.26 6 0.80 0.455 

Student | READ 180 2.62 2.14 565 1.22 0.221 
READ 180*MCLA -3.55 3.01 565 -1.18 0.240 
ELL -7.55 5.51 565 -1.37 0.171 
TCAP Social Studies 06 0.29 0.05 565 5.58 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 13.72 0.052 

Student | Level 1 293.29 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Table I-15 Final Model* for Estimating Fixed and Random Effects for Determining 


READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS Total Reading 


Scores—Year 1, Seventh Grade 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-16 Final Model* for Estimating Fixed and Random Effects for Determining 
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READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS 
Comprehension Scores—Year 1, Seventh Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 188.34 2.06 3 91.28 0.000 
MCLA 1.30 3.05 3 0.48 0.698 
Percent African American 2.93 1.19 3 2.47 0.080 
Percent ELL 4.08 1.78 3 2.30 0.095 
Percent FRL -1.48 0.46 3 -3.22 0.070 

Student | READ 180 -3.12 2.47 420 -1.26 0.208 
READ 180*MCLA 2.96 3.53 420 0.84 0.404 
Gender 2.69 1.75 420 1.54 0.123 
ITBS Comprehension 06 0.34 0.05 420 7.32 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 8.83 0.025 

Student | Level 1 345.99 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 


Table I-17 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS Vocabulary 
Scores—Year 1, Seventh Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 198.84 1.61 6 123.21 0.000 
MCLA -1.40 2.26 6 -0.62 0.559 

Student | READ 180 -1.58 2.47 427 -0.64 0.524 
READ 180*MCLA -1.65 3.54 427 -0.47 0.642 
ITBS Vocabulary 06 0.36 0.04 427 8.46 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 1.53 0.004 

Student | Level 1 377.19 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-18 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Reading/LA 
Scores—Year 1, Seventh Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 497.19 1.90 5 261.90 0.000 
MCLA 0.47 2.64 5 0.18 0.867 
Percent FRL -1.25 0.47 5 -2.67 0.044 

Student | READ 180 -3.02 2.68 562 -1.13 0.261 
READ 180*MCLA 7.79 3.86 562 2.02 0.044 
ELL -8.49 3.53 562 -2.41 0.017 
TCAP Reading 06 36 0.04 562 9.58 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 7.80 0.013 

Student | Level 1 599.19 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Table I-19 Final Model* for Estimating Fixed and Random Effects for Determining 


READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Mathematics 


Scores—Year 1, Seventh Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 508.60 3.89 6 130.64 0.000 
MCLA -12.03 5.49 6 -2.19 0.070 

Student | READ 180 -2.74 3.27 562 -0.84 0.403 
READ 180*MCLA -1.60 4.70 562 -0.34 0.733 
Gender 6.28 2.28 562 2.75 0.007 
TCAP Math 06 0.58 04 562 15.48 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Intercept 79.05 0.072 

Student | Level 1 1024.75 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 
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Table I-20 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Science 
Scores—Year 1, Seventh Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 187.75 1.76 3 106.72 0.000 
MCLA -8.55 2.84 3 -3.01 0.067 
Percent African American -1.91 1.07 3 -1.78 0.169 
Percent ELL -3.58 1.55 3 -2.30 0.094 
Size 0.01 0.01 3 2.37 0.088 

Student | READ 180 -4.02 2.27 535 -1.77 0.077 
READ 180*MCLA 3.55 3.22 535 1.10 0.271 
FRL 4.12 2.49 535 1.65 0.099 
ELL -9.33 3.82 535 -2.44 0.015 
TCAP Science 06 0.23 0.04 535 5.25 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 1.95 0.005 

Student | Level 1 357.42 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Table I-21 Final Model* for Estimating Fixed and Random Effects for Determining 


READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Social Studies 


Scores—Year 1, Seventh Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 184.84 1.73 5 107.09 0.000 
MCLA -2.35 2.46 5 -0.10 0.384 
Percent FRL -1.25 0.47 5 -2.65 0.045 

Student | READ 180 0.39 1.83 533 0.21 0.834 
READ 180*MCLA 2.28 2.60 533 0.88 0.380 
TCAP Social Studies 06 0.15 0.03 533 4.81 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 13.13 0.054 

Student | Level 1 229.26 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-22 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS Total Reading 
Scores—Year 1, Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 205.06 2.38 4 86.33 0.000 
MCLA -3.94 3.17 4 -1.24 0.282 
Percent Female 1.83 0.84 4 2.17 0.092 
Percent ELL 1.21 0.39 4 3.12 0.042 

Student | READ 180 -1.72 2.67 327 -0.64 0.521 
READ 180*MCLA 3.50 3.67 327 0.95 0.342 
ELL -9.07 4.37 327 -2.08 0.038 
ITBS Reading Total 06 0.36 0.06 327 5.98 0.000 

Random Effects 

Level Variance Components Variance icc ¢ 

School Intercept 8.75 0.029 

Student | Level 1 293.19 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-23 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS 
Comprehension Scores—Year 1, Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 199.90 2.83 4 70.54 0.000 
MCLA -8.87 3.78 4 -2.35 0.075 
Percent Female 2.83 1.00 4 2.82 0.049 
Percent ELL 1.47 0.46 4 3.16 0.042 

Student | READ 180 -3.79 3.20 328 -1.19 0.237 
READ 180*MCLA 8.89 4.39 328 2.03 0.043 
Gender 3.19 2.19 328 1.46 0.146 
African American 32.26 19.80 328 1.63 0.104 
Hispanic 46.87 21.11 328 2.22 0.027 
ELL -23.01 8.27 328 -2.78 0.006 
ITBS Comprehension 06 0.35 0.06 328 6.12 0.000 

Random Effects 

Level Variance Components Variance icc ¢ 

School Intercept 19.73 0.044 

Student | Level 1 432.82 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-24 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 ITBS Vocabulary 
Scores—Year 1, Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 215.97 3.65 3 59.18 0.000 
MCLA -10.07 5.68 3 -1.77 0.170 
Percent Female 2.74 1.44 3 1.91 0.147 
Percent FRL 1.49 0.81 3 1.83 0.161 
Size 0.02 0.01 3 2.25 0.100 

Student | READ 180 0.61 3.42 333 0.18 0.859 
READ 180*MCLA -2.62 4.63 333 -0.57 0.571 
Gender -3.26 2.29 333 -1.43 0.155 
ELL -7.94 5.51 333 -1.44 0.150 
ITBS Vocabulary 06 0.28 0.06 333 4.59 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Intercept 0.14 0.000 

Student | Level 1 454.56 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Appendix |. MSRP Evaluation Report, Year 4 March 1, 2011. Page 18 of 30 


Table I-25 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Reading/LA 
Scores—Year 1, Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 519.01 4.28 4 121.32 0.000 
MCLA -21.46 5.98 4 -3.58 0.035 
Percent Female 4.27 1.40 4 3.05 0.043 
Percent FRL 1.91 0.87 4 2.21 0.089 

Student | READ 180 -5.28 4.15 455 -1.27 0.204 
READ 180*MCLA 5.82 5.56 455 1.05 0.297 
Gender 7.11 2.77 455 2.56 0.011 
African American 40.40 29.63 455 1.36 0.174 
Hispanic 56.72 31.29 455 1.81 0.070 
FRL -6.45 4.19 455 -1.54 0.124 
ELL -24.30 11.30 455 -2.15 0.032 
TCAP Reading 06 0.32 0.05 455 5.78 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Intercept 10.80 0.011 

Student | Level 1 444.31 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-26 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Mathematics 
Scores—Year 1, Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 516.83 4.76 4 108.67 0.000 
MCLA -16.02 6.97 4 -2.30 0.080 
Percent Female 3.30 1.69 4 1.96 0.119 
Percent FRL 1.93 1.03 4 1.87 0.133 

Student | READ 180 2.12 4.16 455 0.51 0.609 
READ 180*MCLA -6.31 5.60 455 -1.12 0.261 
African American 167.98 28.70 455 5.85 0.000 
Hispanic 188.46 30.35 455 6.21 0.000 
ELL -16.80 11.12 455 -1.51 0.131 
TCAP Math 06 0.45 0.03 455 13.78 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Intercept 53.29 0.042 

Student | Level 1 1219.24 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 
" The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Table I-27 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Science 
Scores—Year 1, Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 187.61 1.49 125.51 0.000 
MCLA 0.70 1.85 5 0.38 0.721 
Size -0.02 .00 -6.74 0.000 

Student | READ 180 -1.50 2.02 433 -0.74 0.458 
READ 180*MCLA -0.23 2.67 433 -0.09 0.932 
Gender -2.10 1.34 433 -1.57 0.117 
TCAP Science 06 0.18 0.04 433 4.94 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Intercept 25.91 0.115 

Student | Level 1 200.01 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 
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Table I-28 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Impact in MCLA Treatment and Control Schools on 2007 TCAP Social Studies 
Scores—Year 1, Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 186.14 2.56 6 72.63 0.000 
MCLA -2.18 3.50 6 -0.62 0.556 

Student | READ 180 -1.22 2.18 433 -0.56 0.574 
READ 180*MCLA 2.31 2.97 433 0.78 0.437 
African American 27.89 14.49 433 1.93 0.054 
Hispanic 20.95 15.05 433 1.39 0.165 
TCAP Social Studies 06 0.30 0.05 433 6.27 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Intercept 14.24 0.060 

Student | Level 1 224.56 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Table I-29 Final Model* for Estimating Fixed and Random Effects for Determining 


READ 180 One-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
ITBS Total Reading Scores—Sixth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 184.56 1.04 5 177.11 0.000 
MCLA -0.27 1.44 5 -0.19 0.858 
Percent SPED -0.58 0.20 5 -2.83 0.038 

Student | READ 180 -1.57 1.43 492 -1.10 0.274 
READ 180*MCLA 1.64 2.00 492 0.82 0.411 
FRL 2.08 2.25 492 0.93 0.354 
ITBS Total Reading 07 0.45 0.04 492 11.24 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Level 2 0.04 0.017 

Student | Level 1 119.47 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 
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Table I-30 Final Model* for Estimating Fixed and Random Effects for Determining 
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READ 180 One-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
ITBS Comprehension Scores—Sixth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 182.79 1.37 6 133.37 0.000 
MCLA -1.91 1.91 6 -1.00 0.357 

Student | READ 180 -2.63 1.90 497 -1.40 0.165 
READ 180*MCLA 1.65 2.66 497 0.62 0.534 
Gender 2.32 1.30 497 1.79 0.073 
ITBS Comprehension 07 0.40 0.04 497 9.50 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Level 2 1.29 0.005 

Student | Level 1 207.25 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 


Table I-31 Final Model* for Estimating Fixed and Random Effects for Determining 


READ 180 One-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
ITBS Vocabulary Scores—Sixth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 187.18 1.35 5 138.53 0.000 
MCLA 0.33 1.88 5 0.17 0.869 
Percent SPED -0.94 0.27 5 -3.54 0.022 

Student | READ 180 -1.71 1.87 502 -0.92 0.359 
READ 180*MCLA 2.31 2.59 502 0.89 0.372 
FRL 4.71 2.89 502 1.63 0.103 
ITBS Vocabulary 07 0.33 0.04 502 8.29 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Level 2 0.02 0.012 

Student | Level 1 204.91 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 


Appendix |. 


MSRP Evaluation Report, Year 4 


Table I-32 Final Model* for Estimating Fixed and Random Effects for Determining 


March 1, 2011. Page 22 of 30 


READ 180 One-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
TCAP Reading/LA Scores—Sixth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 491.35 2.55 3 193.01 0.000 
MCLA 6.54 3.42 3 1.91 0.146 
Percent African American -1.41 0.45 3 -3.12 0.069 
Percent SPED 2.24 0.97 3 2.31 0.094 
Percent FRL -1.88 1.06 3 -1.77 0.171 

Student | READ 180 -0.44 3.21 658 -0.14 0.892 
READ 180*MCLA 4.44 4.45 658 1.00 0.318 
Gender 7.04 2.17 658 3.25 0.002 
FRL 6.33 4.86 658 1.30 0.194 
TCAP Reading 07 0.47 0.05 658 10.12 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Level 2 0.11 0.003 

Student | Level 1 757.07 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
" The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 


Table I-33 Final Model* for Estimating Fixed and Random Effects for Determining 


READ 180 One-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
TCAP Mathematics Scores—Sixth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 501.90 3.61 5 139.02 0.000 
MCLA -5.18 5.30 5 -0.98 0.374 
Percent SPED 1.95 0.78 5 2.50 0.053 

Student | READ 180 -6.13 2.79 662 -2.20 0.028 
READ 180*MCLA 6.20 3.96 662 1.57 0.118 
TCAP Math 07 0.62 0.04 662 16.04 0.000 

Random Effects 

Level Variance Components Variance Icc* 

School Level 2 30.80 0.035 

Student | Level 1 591.59 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
" The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 
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Table I-34 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 One-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
TCAP Science Scores—Sixth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 189.51 1.42 3 133.55 0.000 
MCLA -5.31 2.08 3 -2.55 0.075 
Percent African American -1.68 0.77 3 -2.17 0.108 
Percent SPED 0.61 0.28 3 2.21 0.104 
Percent ELL -2.29 1.11 3 -2.06 0.123 

Student | READ 180 -1.84 1.74 650 -1.05 0.292 
READ 180*MCLA 2.93 2.40 650 1.22 0.223 
Gender -1.55 1.16 650 -1.33 0.184 
TCAP Science 07 0.33 0.03 650 9.49 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Level 2 0.03 0.001 

Student | Level 1 216.64 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-35 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 One-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
TCAP Social Studies Scores—Sixth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 183.43 2.58 3 71.06 0.000 
MCLA 4.40 3.91 3 1.13 0.343 
Percent African American -0.91 0.54 3 -1.69 0.186 
Percent SPED 2.10 1.11 3 1.88 0.150 
Percent FRL -2.15 1.30 3 -1.66 0.194 

Student | READ 180 -0.75 1.97 640 -0.38 0.703 
READ 180*MCLA -2.83 2.76 640 -1.03 0.306 
Gender -2.92 1.32 640 -2.20 0.028 
FRL -3.67 3.02 640 -1.21 0.226 
ELL -12.70 5.85 640 -2.17 0.030 
African American -14.59 5.45 640 -2.67 0.008 
TCAP Social Studies 07 0.41 0.05 640 8.67 0.000 

Random Effects 

Level Variance Components Variance icc? 

School | Level 2 11.93 0.031 

Student | Level 1 275.04 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-36 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Two-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
ITBS Total Reading Scores—Seventh and Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 195.12 1.39 3 140.15 0.000 
MCLA 4.87 1.96 3 2.48 0.079 
Percent African American -0.47 0.25 3 -1.87 0.154 
Percent SPED 0.90 0.55 3 1.65 0.194 
Percent FRL -1.95 0.62 3 -3.14 0.069 

Student | Grade 8 in Year 2 4.34 1.27 648 3.43 0.001 
READ 180 2.10 1.78 648 1.18 0.239 
READ 180*MCLA -3.30 2.58 648 -1.28 0.201 
Gender 1.63 1.23 648 1.32 0.188 
ITBS Total Reading 06 0.49 0.04 648 11.65 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Level 2 0.32 0.016 

Student | Level 1 246.62 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-37 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Two-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
ITBS Comprehension Scores—Seventh and Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 193.53 1.62 4 119.20 0.000 
MCLA 0.37 2.39 4 0.16 0.885 
Percent African American -0.42 0.21 4 -1.97 0.117 
Percent Female 2.12 0.65 4 3.24 0.041 

Student | Grade 8 in Year 2 2.85 1.54 656 1.85 0.064 
READ 180 0.88 2.17 656 0.41 0.685 
READ 180*MCLA -3.54 3.13 656 -1.13 0.260 
Gender 2.85 1.53 656 1.86 0.063 
ITBS Comprehension 06 0.39 0.05 656 8.64 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Level 2 0.11 0.014 

Student | Level 1 385.32 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 


Table I-38 Final Model* for Estimating Fixed and Random Effects for Determining 


READ 180 Two-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
ITBS Vocabulary Scores—Seventh and Eighth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 200.12 1.56 5 128.15 0.000 
MCLA 2.64 2.21 5 1.20 0.286 
Percent FRL -1.27 0.40 5 -3.18 0.029 

Student | Grade 8 in Year 2 7.45 1.66 664 4.48 0.000 
READ 180 3.65 2.28 664 1.60 0.110 
READ 180*MCLA -3.70 3.29 664 -1.12 0.262 
Hispanic 12.64 6.05 664 2.09 0.037 
ELL -7.94 6.04 664 -1.31 0.189 
ITBS Vocabulary 06 0.31 0.04 664 7.37 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Level 2 0.18 0.012 

Student | Level 1 428.29 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 
’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 


are based on the final model results displayed in the table. 
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Table I-39 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Two-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
TCAP Reading/LA Scores—Seventh and Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 506.72 2.35 2 215.41 0.000 
MCLA -2.19 3.50 2 -0.62 0.596 
Percent African American -1.21 0.36 2 -3.41 0.182 
Percent Female 2.88 1.00 2 2.89 0.107 
Percent SPED 1.93 0.79 2 2.45 0.088 
Percent FRL -1.91 1.04 2 -1.84 0.187 

Student | Grade 8 in Year 2 16.47 1.76 802 9.34 0.000 
READ 180 2.40 2.50 802 99 0.339 
READ 180*MCLA -2.87 3.67 802 -0.78 0.434 
Gender 3.58 1.76 802 2.03 0.042 
ELL -9.99 3.59 802 -2.79 0.006 
TCAP Reading 06 0.244 0.04 802 6.89 0.000 

Random Effects 

Level Variance Components Variance icc * 

School Level 2 0.09 0.021 

Student | Level 1 617.87 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-40 Final Model* for Estimating Fixed and Random Effects for Determining 
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READ 180 Two-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
TCAP Mathematics Scores—Seventh and Eighth Grade 


Fixed Effects 


Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 516.98 3.65 5 141.56 0.000 
MCLA -9.70 5.43 5 -1.79 0.133 
Percent ELL -1.29 0.78 5 -1.67 0.156 

Student | Grade 8 in Year 2 4.41 2.09 805 2.10 0.035 
READ 180 2.63 2.95 805 0.89 0.373 
READ 180*MCLA -7.02 4.33 805 -1.62 0.105 
Gender 4.17 2.08 805 2.01 0.045 
TCAP Math 06 0.56 0.04 805 15.49 0.000 

Random Effects 

Level Variance Components Variance icc * 

School | Level 2 29.96 0.060 

Student | Level 1 860.62 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 


MCLA, and all covariates with p<0.2. 


’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-41 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Two-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
TCAP Science Scores—Seventh and Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 185.76 1.29 3 143.53 0.000 
MCLA -2.37 1.88 3 -1.26 0.296 
Percent Female 1.97 0.52 3 3.76 0.065 
Percent SPED 0.52 0.26 3 2.01 0.130 
Percent ELL 0.45 0.27 3 1.66 0.192 

Student | Grade 8 in Year 2 -2.81 1.12 778 -2.51 0.012 
READ 180 -0.18 1.60 778 -0.11 0.920 
READ 180*MCLA 0.58 2.28 778 0.25 0.800 
Gender -1.63 1.12 778 -1.46 0.145 
African American -6.42 4.06 778 -1.58 0.114 
ELL -6.38 4.25 778 -1.50 0.134 
TCAP Science 06 0.32 0.03 778 9.18 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Level 2 0.04 0.027 

Student | Level 1 240.07 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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Table I-42 Final Model* for Estimating Fixed and Random Effects for Determining 
READ 180 Two-Year Impact in Year 2 in MCLA Treatment and Control Schools on 2008 
TCAP Social Studies Scores—Seventh and Eighth Grade 


Fixed Effects 
Level Effect Estimate | St Error DF t-value Pr > |t| 

School Intercept 188.32 0.97 3 194.29 0.000 
MCLA -3.75 1.43 3 -2.63 0.071 
Percent African American -1.20 0.60 3 -2.00 0.132 
Percent SPED -0.45 0.20 3 -2.24 0.101 
Percent ELL -2.44 0.85 3 -2.88 0.067 

Student | READ 180 -1.40 1.24 775 -1.13 0.259 
READ 180*MCLA 0.28 1.79 775 0.17 0.876 
Hispanic 4.85 1.94 775 2.50 0.013 
TCAP Social Studies 06 0.14 0.02 775 5.83 0.000 

Random Effects 

Level Variance Components Variance icc? 

School Level 2 0.02 0.035 

Student | Level 1 145.86 


* Final model includes the treatment effects of READ 180 and MCLA, the cross-level interaction of READ 180 and 
MCLA, and all covariates with p<0.2. 

’ The interclass correlation coefficient was calculated using a model with no covariates. The variance components 
are based on the final model results displayed in the table. 
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RS 


Principal/Assistant Principal Interview Guide (Cohort II Schools) 
Spring 2010 


Hello, this is from in , . We’ve 
contracted with Memphis City Schools to conduct an independent evaluation of the Memphis 
Striving Readers Program. As part of this four-year evaluation, we are seeking to describe your 
school’s reading practices. You have been asked to participate in this interview because of your 
knowledge about your school as its principal or assistant principal. 


This interview should take approximately 50 minutes. Please answer the questions as best as 
you can. As indicated in the informed consent form that you signed, I would like to audiotape 
this interview so that I may document your responses the most accurately. Only evaluators will 
listen to the audiotape and have access to a transcription of our conversation. Your responses 
will be kept confidential and you will never be identified by name when we report the results of 
these interviews. Your identity will never be revealed to program staff or any individual 
associated with your school or school district. While researchers may report quotes collected 
during interviews with school principals, every effort possible step will be taken to ensure your 
confidentiality. RBS and Edvantia staff members will store this information in a secure 
location.’ You are free to stop participating or withdraw at any time. Let me know if you would 
like to skip a question because you do not know how to respond to it. 


Do I have permission to audiotape this interview? May I start the interview now? 
1. How long have you been in your current position as principal/assistant principal? 
2. Do you have any prior experience with literacy instruction? Please describe. 


3. What is your understanding of the Memphis Content Literacy Academy (MCLA) and its 
purpose? 


4. Please describe any specific links (or disconnects) between MCLA and current school 
improvement plans. 


5. To what extent, if at all, have you been involved in MCLA since school started this year? 
Did you and/or designated administrative staff attend the kickoff? [Probe: Who 
attended? | 
Did you and/or designated administrative staff attend the first principal fellowship 
meeting? [Probe: Who attended? ] 


' Data collected for research purposes are stored in compliance with ISO 17799 requirements for access, security, and 
redundancy. Data are stored in an encrypted format in centralized, electronically and physically secure servers at RBS and 
Edvantia for a period not to exceed five years. All electronic data of a personal nature are safeguarded and available only 
to those project leaders, staff, and technologists having a need to know within the specific criteria as set forth in the 
approved project plan. The Edvantia Institutional Review Board has the authority to inspect consent records and data files 
only to assure compliance with approved procedures. 
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Who at this school handled the rostering of students randomly assigned to receive READ 
180? 

How many Principal Fellowship meetings did you attend this year? Which other staff, if 
any, also attended these sessions? 


6. What did you expect teachers to be able to do as a result of their participation in MCLA? 


7. Do you think that MCLA requires teachers to do different things in addition to what is already 
expected of them? [Probe: If YES, please describe whether the additional demands support or 
conflict with achievement of other/more important priorities. ] 


8. How important was it for you to communicate to teachers the expectation that they work with 
their coach to implement MCLA strategies? [Probe: In what settings, if any, and how often, if at 
all, would you this occured? Please describe. ] 


9. What were your expectations of the principal fellowship? [Probe: What, specifically, did you 
expect to learn? What supports, if any, were you expecting the fellowship to provide? | 


10. To what extent do you (or your designee) mention the importance of literacy in the content 
areas in daily school announcements? [Probe: Would you say once a week, every other week, 
every now and then? What was the nature of those comments? Can you give me an example? ] 


11. How often, if at all, has MCLA been discussed at faculty meetings? [Probe: How frequent 
are faculty meetings at your school? Is MCLA discussed at each meeting? What in particular is 
discussed? ] 


12. In what ways, if at all, have you or your staff sought out [or reallocated] additional resources 
related to literacy instruction? 


13. Which MCLA strategies, if any, have you seen “in action?” [Probe: What do you notice 
when you see MCLA in action? ] 


14. To what extent, if at all, did your school’s schedule enable grade-level teachers in a 
department to work together on integrating literacy? [Probe: How frequently did these meetings 
occur? How long were these meetings? ] 


15. Since the start of the 2009-2010 school year, how often if at all, did you or designated staff 
conducted walkthroughs content area classes? What was the purpose of those walkthroughs? 
What did you look for when you observed part of a lesson? 


16. How realistic was it to expect you to conduct walkthroughs and observations of MCLA 
teachers’ literacy strategy implementation? [Probe: How often?] 


17. To what extent did you (or designated staff) provide feedback to teachers about observed use 
of literacy strategies? [Probe: How soon after the observation was feedback provided? What, 
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specifically, did you provide feedback about? Did you comment on teachers' use of strategies, 
students' use of strategies, both, or neither? ] 


18. Who was your designated MCLA literacy coach? What were your expectations for her? In 
what ways, if any, did you interact with her? Did you see her as a resource for providing school- 
wide support? [Probe: Was she a member of any school-wide teams you assembled? Did you 
expect her to provide in-service professional development for school staff? Did you seek her 
input on decisions related to literacy? Ask for supporting details/examples.] {NOTE: Interviewer 
should let respondent name teams rather than mention leadership team specifically.} 


19. School Improvement Goals: What were your school’s main student achievement 
improvement goals for 2009-2010? [Probe: Was MCLA mentioned in the school’s improvement 
plan?] 


20. How were these goals identified? [Probes: Internally, by the principal/school staff? Under 
NCLB as a result of performance on adequate yearly progress (AYP) indicators? ] 


21. How much teacher turnover was there this year? 


Thank you so much for your time today. 
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Researchers are continuing to study a Striving Readers Project involving your school. Please take a few minutes 
to complete this final survey even if you have done so previously. To protect your identity, only researchers will 
have access to surveys and only group-level results will be analyzed and reported. Thank You! 


Please fill in the best response for each item below. 


Name (First, Last): 


Last 6 digits of your social security number: 


1. Gender: © Male O Female 


2. Race/Ethnicity: 


© Asian/Pacific Islander 

Black, non-Hispanic 

Hispanic 

Native American/Alaska Native 
White, non-Hispanic 


Multi-racial 


>. or 1 ay a 


OQ 20-29 O 30-39 O 40-49 
O 50-59 © 60-69 O 70+ 


4. Current level of education: 


Bachelor’s degree 

Bachelor’s degree + 15 or more credits 
Master’s degree 

Master’s degree + 15 or more credits 


Educational Specialist (Ed.S.) 


CD) Ss SO 


Doctorate 


Other (specify: ) 


(Like this: @ Not like this: (Y Y ~*~) 


5. Current job title: 


O 


OO O 


WwW 
O 
O 
© 
O 
O 
O 
'@. 


Full-time teacher 
Part-time teacher 
Literacy coach 


Other (specify: ) 


hich Tennessee teacher license do you hold? 


Alternative license 
Apprentice teacher license 
Interim license 

Out-of-state teacher license 
Professional teacher license 
None 


Other (specify: ) 


7. Are you licensed (professional or apprentice) 
in the grade & subject you currently teach? 


O- Yes O No 


8. Where have you taught this year? 


O 


O 
O 
O 


A. Maceo Walker © Hickory Ridge 


American Way © Lanier 
Corry O Riverview 
Hamilton © Sherwood 


9. Did you attend Memphis City public schools as 
a student? 


O- Yes O No 


> Please complete next page > 
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10. Please bubble the subject(s) and grade level(s) you 


currently teach. 


English/Language Arts 


Social Studies 


Special Education 


Ll; 


OOO 


{2 


3. 


How many class periods do you teach on a 
typical day? 

none © one © two 

three © four © five 

Six Or more 


How many years have you been teaching 
FULL TIME at your current school? 

I’m not, or I have never been full time 
0-2 years 

3-5 years 

6-10 years 

11-20 years 

21-30 years 


oe Or a OD 


More than 30 years 


This past school year (2009-2010), how often 
would you say you helped your students select 
CRC materials appropriate for their reading 
level? 

© Less than once per month 

About once a month 


A few times a month 


About 1-2 days per week 


OO OO 


At least 3 days per week 


March 1, 2011. Page 2 of 4 
14. If you answered ABOUT ONCE A MONTH OR 
MORE FREQUENTLY in Ques. #13, in what 
percentage of your class periods did you help students 
select materials at their reading level? 


O Less than 25% 


CO 25% 

CO 25-49% 
O 50% 

O. Fa% 

O 100% 


15. Considering all of the professional development 
you had THIS SCHOOL YEAR (2009-2010) 
(excluding pre-service training), approximately how 
many total hours did you spend in activities in which 
the following subjects were a major focus: 


More 

than 

1-8 9-32 32 

None Hours’ Hours’ Hours 


New methods of teaching 
(e.g. cooperative learning) 


Integration of educational 


technology into the O O © O 
classroom 


Classroom management, 
including student discipline 


O O O O 


Addressing the needs of 
students with disabilities ©) O O O 


> Please complete next page > 
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16. To what extent do you feel prepared to engage in the activities below, and how often 
(if at all) did you engage in those activities during the school year (2009-10)? 


Please check a box on the left that best represents how prepared you feel currently and the 
box on the right that reflects how often you used this technique during the 2009-2010 


2 

a 

jo) 

a 
z 8 school year. Please fill only one bubble per side for each item below. 
s 3 = 2 Preparedness Frequency cl 
o| & ae) 1 — Not at all prepared 1 — Never s 
a ces a. s 2—A little prepared 2 — Rarely 5) 2 
S\o/ el] sé 3 3 — Prepared 3 — Sometimes a & + 
= = 8. Sl ¢ 4 — Well prepared 4 — Often a e 
zlel/felels 5 — Can teach others to do this 5 — Almost always 7 elalo = 
GO) Os Ono - Have students read aloud from core subject area texts and/or supplemental texts daily for Cm eer iron 

at least five minutes per period 
O O O O ©. »b. Identify “bridging books” (part story and part information) Ce 
O O O O Oc. Askhigher order questions and require students to justify their answers OL ee) 
O O OO O Od. Pre-test students before the beginning of a new unit of instruction O ©. O GO © 
O O O O Oe. _ Discuss and analyze new vocabulary before reading See Sa Sees e 
O O O O O | f. — Show relationships of words/concepts using graphic organizers or thinking maps OOO © © 
O O O O O- g Create, elaborate, and sort subject-related vocabulary word lists ey ay 1) 
O O O O O  h.. Establish the purpose(s) for reading a text selection Ch Xo 4D ip 
O O O O © i. Have students read in pairs Cm Ore One re 
COO Oo Oo F Model for students, and provide guided practice with feedback on oral retelling strategies SOG o 6 
of selected subject area texts 

O O O O Ok. Model use of thinking maps to construct written summaries of selected text ee S a Ore oe) 
O O O O O 1. _ Link students’ background knowledge and experiences to new vocabulary/concepts O Cr 2-0 


> Please complete next page > 
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16. Continued ... 


‘a Please check a box on the left that best represents how prepared you feel currently and the 
a box on the right that reflects how often you used this technique during the 2009-2010 
a school year. Please fill only one bubble per side for each item below. 
oO 
z 2 
c 3 5 Preparedness Frequency - 
& | & a = 1— Not at all prepared 1 — Never i 
| 5 S| 2 —A little prepared 2 — Rarely 2 2 
a| |B] e 3 3 — Prepared 3 — Sometimes & < 
e|/el] é = = 4 — Well prepared 4 — Often o = 2 . g 
> et oD OD ‘ iI ~~ Z| 
F zlzele/é 5 — Can teach others to do this 5 — Almost always o S186 a 
O O ©O © O mm. Model new learning strategies for students OC) © 
O O OC © © n._ Differentiate instruction using multi-leveled materials GG 2 0 GO © 
O O © © Oo. Teach students to ask questions, before, during, and after reading text selections CoP Cm Sye Oa @ 
OOO. © 6 a Provide guided practice for students trying out new learning skills with peer or teacher Ooo Oo 6G 6 
feedback 
O O OO O OO  gq.. Provide instruction on the different forms of writing found in content area textbooks Cre Oo OS 
S66 6 & < Offer small group instruction and practice several times per week according to students Cmemcmecis 
achievement levels in reading 
O O O O Os. Use the writing process as part of content learning OO Oe 
O O OO O © t.. Adapt instruction for students having special needs QO O 0 OG 0 
O O O O Ou. Use cooperative learning groups A) DAO 
O O O O Oy. Use oral reading (whole class/small group) in subject area materials O-O 0 0 © 
ons mes mcs me Use benchmark assessment data to identify students’ achievement of content-specific cuvien sano. 
learning goals 
Go 6: o 6 o& « Use direct, explicit instruction when teaching new reading/study skills related to my core Oo OG o-o 


subject area 


Thank you! 
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epUCATIGg 
<S G se Ova, 
» 


College of Education 

The University of Memphis 

Instruction and Curriculum Leadership 

RDNG 8543: Advanced Reading Instruction/ 


Z b 


LEADERSHIP 


) i Special Learners, Focus on Improving Academic 
Vocabulary, Comprehension & Fluency 
Fall 2009 


COMMITMENT TO DIVERSE COMMUNITIES 


University of Memphis College of Education 


College of Education Norms 


I take 100% responsibility. 

I seek equity of voice. 

I am willing to talk about sensitive issues. 

I listen for understanding. 

I appreciate the strengths and contributions of others. 
I bring positive energy and encouragement to the team. 
I commit to the mission of the college. 


ICL 7008: Seminar in Curriculum Improvement: Focus on Subject Area Vocabulary, 
Comprehension & Fluency Learning 


Course Description: 

The Memphis Content Literacy Academy is a practice-oriented course that explores 1) knowledge of 
relevant research involving urban populations, 2) essential skills and knowledge to learned, 3) evidence- 
based teaching practices, and 4) ways of adapting instruction to meet special student needs. Unlike other 
graduate courses, we will study a relative few concepts in depth and apply them in participants’ own 
classrooms with the assistance of a Literacy Coach (LC). 


Class Meetings: Time and Locations 
Location for all classes: American Way Middle School 


SCIENCE TEACHERS 
Tuesdays, 4:15-7:15 P.M. 


MATHEMATICS, LANGUAGE ARTS, & SOCIAL STUDIES TEACHERS 
Thursdays, 4:15-7:15 P.M. 


Suggested Text(s): 
Marzano, R.J., & Pickering, D.J. (2005). Building academic vocabulary: Teacher’s Manual. Alexandria, 
VA: Association for Supervision and Curriculum Development. 


*These are provided free to you by the Striving Readers grant. PLEASE BRING THESE AND A TEACHER’S 
EDITION (TE) FROM ONE OF YOUR TEXTBOOKS USED FOR YOUR CLASSES TO EACH MEETING. 


Other available resources: 
¢ A “Curriculum Resource Center (CRC)” is located at your school this semester to assist you with 
your daily classroom instruction. 
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Memphis Content Literacy Academy/Striving Readers Project 


Support of Conceptual Framework: 

This is the final of four semesters in a study of evidence-based literacy strategies that may be applied in 
core subject area and special education classrooms. This course is a major part of the Memphis Striving 
Readers Project, a federally funded program. The MCLA is a joint University of Memphis/Memphis City 
Schools venture aimed at helping subject area teachers at specially selected middle schools in MCS develop 
expertise in implementing scientifically-based literacy practices as part of instruction in mathematics, 
science, social studies, and language arts education so that children living at the poverty level can better 
realize their full potential in American education. Memphis is one of only eight experimental Striving 
Readers sites in The United States. Results of our project will be available to help teachers of middle school 
students all over America achieve their potential. 


Course Objectives: 

The objective of this course is to assist practicing teachers in improving middle school students’ 
understanding of textbook readings BEFORE, DURING, and AFTER students read an assignment. We will 
engage in deeper understanding by reviewing and implementing selected strategies in the areas of 
VOCABULARY learning, COMPREHENSION of subject area texts, and increasing students’ READING 
FLUENCY. 


Schedule, Assignments, Exams, and Grading Criteria: 


Date (week) (CAPs)* Critical Information 

Session 1 “Something Old, Something New” 1. Identify a Unit of Study 
for CAP #1 (should be 
one coming up in a few 
weeks). 

2. From the unit you have 
chosen for CAP #1, select 
five (5) “non-negotiable” 
words/concepts you feel 
all your student must 
learn and bring these to 
Session 2. 


September 1, 3 INCREASING VOCABULARY KNOWLEDGE 


PART I: Course Introduction & Syllabus Review 


PART I: TEACHING TACTIC/CONCEPT 
Strategies to help students learn BEFORE-, DURING- 
and AFTER reading academic text assignments 


PART HI: BUILDING VOCABULARY 

“Something Old”: Semantic Feature Analysis (SFA); 
Word Maps (as an “assessment tactic”’) 

“Something New”: Concept Maps; 

6 Degrees of Separation Game 


PART IV: Introducing Classroom Action Plan #1 


Session 2 


Sept. 8, 10 


PART 1: CAP #1 Discussion & demonstrations by the 
instructor on ways to implement CAP | in your core 
subject area classes 


PART II: Homework Application 

Construct a Pre-Test and Post-Test for whole class 
assessment using word map to administer to your 
classes 


PART III: Discuss “homework” assignments 


Fall Semester 2009, The University of Memphis (1081909) -2- 


1. ASSESSMENT TACTIC: 
Administer the “word maps 
pre-test” you constructed in 
class to at least one of your 
classes and complete the 
summary form provided in 
class. Bring your results to 
Session 3. 


2. Bring your best ideas so far 
about implementing CAP #1 
to share and discuss with our 
“community of learners” in 
Session 3. 


RDNG 8543 
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Session 3 


Sept. 15, 17 


Session 4 


Sept. 22, 24 


Session 5 


Sept. 29, Oct. 1 


Fall Semester 2009, The University of Memphis (1081909) 
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Memphis Content Literacy Academy/Striving Readers Project 


PART I: Homework Discussion & Analysis about 
word maps as an assessment tool. Discuss your 
students’ range of knowledge of the five (5) “non- 
negotiable” words/concepts you identified for this unit 
of study 


PART II: OPTIONAL STRATEGY 
Classroom Blogging (with or without technology) 


PART III: In-class work session to prepare for CAP #1 
“Community of scholars” work groups sharing ideas 
for CAP #1 from homework assignment 


PART IV: “Great Ideas” Gallery Walk for CAP #2 
IMPROVING COMPREHENSION 


PART I: “Something New” ASSESSMENT TACTIC 
Marzano’s Question Stems as pre-test/post-test 
assessments 


PART II: “Something New” 
Previewing text readings using SEARCHES or 
THIEVES (BEFORE- Reading) 


Fix-Up Strategies (DURING- and AFTER Reading 
Academic Texts) 


PART III: Introducing Classroom Action Plan #2 
PART IV: Discuss “homework” assignments 


PART I: Discussion & demonstrations by the instructor 
on ways to implement CAP 2 in your core subject area 
classes 


PART I: Homework Application 
Construct in class a Pre-Test and Post-Test for whole 
class assessment using Marzano’s Question Stems 


PART III: Teacher Tactic OPTION 

Presentation on how teachers can do a content analysis 
to identify key facts, concepts, and generalizations in a 
unit of study 


PART IV: Discuss “homework” assignments 


CAP #1 DUE 


Schedule a time with your 
Literacy Coach to review your 
plans for CAP #1. 


1. Identify a Unit of Study for 
CAP #2 (should be one 
coming up in a _ few 
weeks) and bring that 
information and texts you 
plan to use to Session 5. 


2. ASSESSMENT TACTIC 
assignment: Identify 5-10 
important facts and/or 
concepts from the unit 
you have identified to use 
in implementing CAP #2 
and bring this information 
to Session 5 (a summary 
form will be provided in 
class for this task). 

1. ASSESSMENT TACTIC: 

Administer the Marzano’s 

Question Stems pre-test you 

constructed in class to at least 

one of your classes and 
complete the student summary 
form provided in class. Bring 

your results to Session 6. 


2. Bring your best ideas so far 
about implementing CAP #2 
to share and discuss with our 
“community of learners” in 
Session 6. An outline form is 
to be provided for this 
assignment. 
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Session 6 


Oct. 6, 8 


Session 7 


Oct. 13, 15 


Session 8 


Oct. 20, 22 
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Memphis Content Literacy Academy/Striving Readers Project 


PART I: Homework Discussion & Analysis about 
Marzano’s Question Stems as an assessment tool. 
Discuss your students’ range of knowledge of the five 
(5) “non-negotiable” words/concepts you identified for 
this unit of study. 


PART II: OPTIONAL STRATEGY 
Wiki Writing (with or without technology) 


PART III: In-class “Community of Scholars” work 
groups sharing ideas for CAP #2 


PART IV: “Great Ideas” Gallery Walk for CAP #2 
BUILDING READING FLUENCY 


PART I: “Something New” ASSESSMENT TACTIC 
“one-minute of academic reading” 


PART II: “Something New” 
Neurological Impress & Scooping 


PART III: Introducing Classroom Action Plan #3 


PART IV: Discuss “homework” assignment 


PART I: Demonstrations by the instructor on ways to 
implement strategies in your core subject area for CAP 
#3 using Neurological Impress & Scooping as well as 
other strategies 


PART II: Using materials from your homework 
assignment, construct a whole class assessment using 


the one-minute of academic reading strategy 


PART III: Discuss “homework” assignment 


Fall Semester 2009, The University of Memphis (1081909) -4- 


Schedule a time with your 
Literacy Coach to review your 
plans for CAP #2. 


1. Identify a Unit of Study for 
CAP #3 (should be one 
coming up in a few 
weeks) and bring that 
information and texts you 
plan to use to Session 8. 


2. ASSESSMENT TACTIC 
assignment: Identify a 
200-word passage from 
the unit of study you 
select for implementing 
CAP #3. You may use a 
reading from your adopted 
textbook or a 
supplemental text. Bring a 
copy of this passage to 
Session 8. 

1. ASSESSMENT TACTIC: 

Administer the one-minute of 

academic reading pre-test 

you constructed in class to at 
least one of your classes and 
complete the student summary 
form provided in class. Bring 
your results to Session 9. 


2. Bring your best ideas so far 
about implementing CAP #3 
to share and discuss with our 
“community of learners” in 
Session 6. An outline form is 
to be provided for this 
assignment. 
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Session 9 In-class work session to prepare for CAP #3 Schedule a time with your 
Literacy Coach to review your 
Oct. 27, 29 PART III: In-class “Community of Scholars” work plans for CAP #3 
groups sharing ideas for CAP #3 


PART IV: “Great Ideas” Gallery Walk for CAP #3 


CAP #2 Due 
* Please note that changes may be made to the course activities and assignments at the discretion of the MCLA 
Leadership Team. 
** All readings and assignments should be completed by the date of the class meeting indicated. 


RequirementsPossible Points 


Classroom Action Plan (CAP): There will be three (3) Classroom Action Plans 60 

(CAP) assigned for you to implement in your classroom. Your Instructional Coach 

will meet with in August or early September to discuss the procedure for 

completing this requirement. The CAPs will also be posted on our website, 
www.memphisstrivingreaders.org. Following are the deadlines for turning in 

your completed CAPs: 

CAP #1 is due by not later than class time on September 29/October 2 (Turn in to your Instructor) 
CAP #2 is due by not later than class time October 27/29 (Turn in to your Instructor) 

CAP #3 is due by not later than November 20 (Turn in to your Literacy Coach at Your School) 


Attendance & Participation: You are expected to attend all class 40 
sessions and participate in “Instructional Conversations (IC)” and 

“Joint Productive Activities (JPA).” Your participation will be 

evaluated each week by your peers and the instructor. 


Grading Scale: A = 93 — 100 points B = 85 — 92 points C=77 — 84 points 


D = 69 — 76 points F = 66 points and below 


Implementing Classroom Action Plans: How the Literacy Coaches Will Assist You At Your School 


Literacy Coaches (LC) are provided at your school primarily to assist you in implementing Classroom 
Action Plans (CAPs), find materials and ideas for your classes, and to help you solve any instructional 
issues you feel will help your students learn. They are also in charge of the new Curriculum Resource 
Center (CRC) at your school that houses supplemental teaching/learning materials for your instruction. LCs 
are never put in the position of serving as a teacher appraiser for MCS or the principal-- they are there to be 
helpful colleague. 


Implementing Classroom Action Plans (CAPs): How the Instructional Coaches Will Assist You At 


Your School (continued) 


In terms of designing and implementing CAPs and your classroom, the ICs will use a routine following 
these steps: 


1, The Classroom Action Plan (CAP) will be introduced in class by your instructor. 
Your Literacy Coach(es) will meet with you at your school to clarify the CAP further, answer 
questions, model the strategy as needed, and schedule times and dates to meet with you for 
future CAP activities. 
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3. After you draft your lesson plans for carrying out the CAP, your LC will meet with you to 
review the lesson plans and provide feedback. They will also confirm a time to watch you 
teach one class session from your plan (called “Teaching Rehearsal”). 

4. The LC will next observe your “Teaching Rehearsal” and provide feedback (Debrief) later 
that day or the next day. This Debrief is intended to be a formative assessment and you will 
not be graded. 


ay After your Debrief with the LC following your Teaching Rehearsal, you will revise your 
lesson plans as needed. 
6. “Performance Teaching” is the final step in implementing your CAP and will determine your 


grade for the CAP. The IC will observe one class session and then Debrief with you again to 
discuss your execution of the plan and provide helpful feedback. 
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Following is a model that shows the CAP Coaching Cycle: 


Final 
Debrief 


Debrief/Revise 
Lesson Plan 


Fall Semester 2009, The University of Memphis (1081909) 


Cap Introduced in 
Class 


Coaches 
Clarify CAP 
& Model 
@ School 


Draft Lesson Plans 
w/Coach’s Feedback 


Teaching 
Rehearsal 
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Other Course Requirements 


Professional Participation: Your active participation in this class is essential for building a productive 
learning community. It is expected that you will give freely of your ideas, constructively react to the ideas 
of others, and offer constructive suggestions for the good of the group. Responsibility for participation also 
includes: completing assignments on schedule, a willingness to take risks in sharing your opinions, and 
verbally participating in class discussions and activities. 


Attendance Requirements for this Course: Class interactions are critical to professional growth and 
development. Class attendance and cooperative engagement in class cannot be duplicated in any other 
way. You are expected to attend all classes for the full time period called for in the schedule. 
Attendance will be documented at each class period and includes coming late to class or leaving early. 
Two late arrivals to class and/or early exits total one absence. Class attendance will be reflected in 
your participation grade. For every absence, beginning with the second absence, five points will be 
deducted from your participation grade earned for each absence. Missing three or more classes 
will result in a failing grade. 


Americans with Disabilities Act: The University of Memphis does not discriminate on the basis of 
disability in the recruitment and admission of students, the recruitment and employment of faculty and 
staff, and the operation of any of its programs and activities, as specified by federal laws and regulations. 
The student has the responsibility of informing the course instructor (at the beginning of the course) of any 
disabling condition, which will require modification to avoid discrimination. Faculty are required by law to 
provide "reasonable accommodation" to students with disabilities, so as not to discriminate on the basis of 
that disability. Student responsibility primarily rests with informing faculty at the beginning of the semester 
and in providing authorized documentation through designated administrative channels. 


Academic Integrity and Student Conduct: 

Expectations for academic integrity and student conduct are described in detail on the website of the Office 
of Student Judicial and Ethical Affairs (http://saweb.memphis.edu/judicialaffairs). Please take a look, in 
particular, at the sections about “Academic Dishonesty,” “Student Code of Conduct and Responsibilities,” 
and “Disruptive Behaviors.” We expect students to be aware of these guidelines and to conduct themselves 
accordingly. 
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The University of Memphis 

Instruction and Curriculum Leadership 

ICL XXXX: Advanced Reading Instruction/ 
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LEADERSHIP 


) i Special Learners, Focus on Improving Academic 
Vocabulary, Comprehension & Fluency 
Spring 2010 


COMMITMENT TO DIVERSE COMMUNITIES 


University of Memphis College of Education 


College of Education Norms 


I take 100% responsibility. 

I seek equity of voice. 

I am willing to talk about sensitive issues. 

I listen for understanding. 

I appreciate the strengths and contributions of others. 
I bring positive energy and encouragement to the team. 
I commit to the mission of the college. 


ICL XXXX: Seminar in Curriculum Improvement: Focus on Subject Area Vocabulary, 
Comprehension & Fluency Learning 


Course Description: 

The Memphis Content Literacy Academy is a practice-oriented course that explores 1) knowledge of 
relevant research involving urban populations, 2) essential skills and knowledge to learned, 3) evidence- 
based teaching practices, and 4) ways of adapting instruction to meet special student needs. Unlike other 
graduate courses, we will study a relative few concepts in depth and apply them in participants’ own 
classrooms with the assistance of a Literacy Coach (LC). 


Class Meetings: Time and Locations 
Location for all classes: American Way Middle School 


SCIENCE TEACHERS 
Tuesdays, 4:15-7:15 P.M. 


MATHEMATICS, LANGUAGE ARTS, & SOCIAL STUDIES TEACHERS 
Thursdays, 4:15-7:15 P.M. 


Support of Conceptual Framework: 

This is the final of four semesters in a study of evidence-based literacy strategies that may be applied in 
core subject area and special education classrooms. This course is a major part of the Memphis Striving 
Readers Project, a federally funded program. The MCLA is a joint University of Memphis/Memphis City 
Schools venture aimed at helping subject area teachers at specially selected middle schools in MCS develop 
expertise in implementing scientifically-based literacy practices as part of instruction in mathematics, 
science, social studies, and language arts education so that children living at the poverty level can better 
realize their full potential in American education. Memphis is one of only eight experimental Striving 
Readers sites in The United States. Results of our project will be available to help teachers of middle school 
students all over America achieve their potential. 
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The objective of this course is to assist practicing teachers in improving middle school students’ 
understanding of textbook readings BEFORE, DURING, and AFTER students read an assignment. We will 
engage in deeper understanding by reviewing and implementing selected strategies in the areas of 
VOCABULARY learning, COMPREHENSION of subject area texts, and increasing students’ READING 


FLUENCY. 


Schedule, Assignments, Exams, and Grading Criteria: 


Spring 2010 
Class Sessions & Dates 


Primary Topics for Professional 
Development 


Assignments & Notes 


Session 1 


January 12, 14 


Semester Outline & Syllabus 
Introducing Anchor Activities 
Review of Content Literacy Assessments 


Introduce Classroom Action Plan (CAP) #1 


1. Identify an upcoming unit of study and 
select reading/text materials. 

2. Select and Begin Administering Three 
Self-Selected Assessments (one each for 
Fluency, Vocabulary, Comprehension) 
after obtaining approval from your 
literacy coach. 


Session 2 


January 19, 21 


Review of Before-During-, After Reading 
Options for Fluency and Vocabulary 
Development 


Analysis of Student Work for Planning 
Instruction 


Anchor Activities Part 2 


Introduce Classroom Action Plan (CAP) #2 


1. Complete Administration of three Self- 
Selected Assessments (one each for 
Fluency, Vocabulary, Comprehension). 

2. Choose B-D-A Activities in at least two 
Categories (Fluency, Vocab., Comp) 
matched to student needs from your 
assessments. 

3. Create/Choose Two Anchor Activities to 
include in your plan. 


Session 3 


January 26, 28 


Review of Before-During-, After Reading 
Options for Comprehension 


Share Your Analysis of student work 


Poster Session #1: Tips 


CAP #1 due at this class meeting with your 
literacy coach’s signature. 


Prepare your first Poster Session that explains 
your implementation plan. 


Share Your Plan in Poster Session #1 & 


Begin Implementation of your Content Literacy 


February 9, 11 


Working with Over-aged Students 


Session 4 Receive Suggestions from your Colleagues Unit (self-selected B-D-A , Anchor Activities, 
and Coordinated Homework Plan) 
February 2, 4 
Homework & Parent Involvement Continue Implementation of your Content 
Session 5 Literacy Unit (self-selected B-D-A , Anchor 


Activities, and Coordinated Homework Plan) 


Session 6 


February 16, 18 


NO CLASS - Continue Implementation of 
your Content Literacy Unit 


In-school Implementation & Scheduled 
Consultation with Literacy Coaches/Course 
Instructor. 
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Memphis Content Literacy Academy/Striving Readers Project 


Session 7 


February 23, 25 


NO CLASS - Continue Implementation of 
your Content Literacy Unit 


Final Week of In-school Implementation & 
Scheduled Consultation with Literacy 
Coaches/Course Instructor. 


Conduct Post-Teaching Assessments & Analysis 
by the end of this week; Compile & Discuss post- 
test results with your literacy coach. 


Tips for Preparing Your Final Poster Session 


CAP #2 due at this class meeting with your 


Session 8 literacy coach’s signature. 
Bring descriptions of your Analysis of Student 
March 2, 4 Work (pre- and post-test results), B-D-A, Prepare for your Poster Session according to 
Anchor Activities, and Coordinated criteria specified in class. 
Homework Plan (from CAP #2) 
Poster Sessions & Final Evaluation 
Session 9 Planning for the Laureate Ceremony 
March 9, 11 
Session 10 Laureate Ceremony 
TBA 


* Please note that changes may be made to the course activities and assignments at the discretion of the MCLA 


Leadership Team. 


** All readings and assignments should be completed by the date of the class meeting indicated. 


Requirements & Possible Points 


Classroom Action Plan (CAP): There will be two (2) Classroom Action Plans 20 
(CAP) assigned for you to implement in your classroom. Your Instructional Coach 
will meet with you to discuss the procedure for completing this requirement. 


Poster Sessions: You will make two poster presentations this semester as a way 50 
of sharing your work in the classroom. Guidelines will be provided. The first 
poster session will count 15 points, and the final poster session will be 


worth 35 points. 


Laureate Ceremony: Our final activity will be the Laureate Ceremony. Each 10 
participant will be asked to display their final poster session at this event. 


Attendance & Participation: You are expected to attend all class 
sessions and participate in Instructional Conversations and 

Joint Productive Activities (JPA). Your participation will be 
evaluated by your peers and the instructor. 


20 


D = 69 — 76 points 


Grading Scale: A = 93 — 100 points B = 85 — 92 points 


F = 66 points and below 


C=77 — 84 points 


Implementing Classroom Action Plans: How the Literacy Coaches Will Assist You At Your School 
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Memphis Content Literacy Academy/Striving Readers Project 


As with past MCLA classes, Literacy Coaches (LC) are provided at your school to assist you in 
implementing Classroom Action Plans (CAPs), help you find materials and ideas for your classes, and to 
help you solve any instructional issues you feel will help your students learn. They are also in charge of the 
Curriculum Resource Center (CRC) at your school that houses supplemental teaching/learning materials for 
your instruction. LCs are never put in the position of serving as a teacher appraiser for MCS or the 
principal-- they are there to be helpful colleague. 


Other Course Requirements 


Professional Participation: Your active participation in this class is essential for building a productive 
learning community. It is expected that you will give freely of your ideas, constructively react to the ideas 
of others, and offer constructive suggestions for the good of the group. Responsibility for participation also 
includes: completing assignments on schedule, a willingness to take risks in sharing your opinions, and 
verbally participating in class discussions and activities. 


Attendance Requirements for this Course: Class interactions are critical to professional growth and 
development. Class attendance and cooperative engagement in class cannot be duplicated in any other 
way. You are expected to attend all classes for the full time period called for in the schedule. 
Attendance will be documented at each class period and includes coming late to class or leaving early. 
Two late arrivals to class and/or early exits total one absence. Class attendance will be reflected in 
your participation grade. For every absence, beginning with the second absence, five points will be 
deducted from your participation grade earned for each absence. Missing three or more classes 
will result in a failing grade. 


Americans with Disabilities Act: The University of Memphis does not discriminate on the basis of 
disability in the recruitment and admission of students, the recruitment and employment of faculty and 
staff, and the operation of any of its programs and activities, as specified by federal laws and regulations. 
The student has the responsibility of informing the course instructor (at the beginning of the course) of any 
disabling condition, which will require modification to avoid discrimination. Faculty are required by law to 
provide "reasonable accommodation" to students with disabilities, so as not to discriminate on the basis of 
that disability. Student responsibility primarily rests with informing faculty at the beginning of the semester 
and in providing authorized documentation through designated administrative channels. 


Academic Integrity and Student Conduct: 

Expectations for academic integrity and student conduct are described in detail on the website of the Office 
of Student Judicial and Ethical Affairs (http://saweb.memphis.edu/judicialaffairs). Please take a look, in 
particular, at the sections about “Academic Dishonesty,” “Student Code of Conduct and Responsibilities,” 
and “Disruptive Behaviors.” We expect students to be aware of these guidelines and to conduct themselves 
accordingly. 
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Classroom Action Plan (CAP) #1 — Fall 2009 


Vocabulary Instruction: BEFORE, DURING, AFTER Reading 


Name School 


Subject Area/Grade Level Date Assigned: September 1/3 


Due: At your class meeting on September 29/October 1 


Directions: Develop lesson plans and execute your CLASSROOM ACTION PLAN (CAP) for 
your academic classes. Note: Please have your Literacy Coach(es) sign and date each stage of 
your CAP implementation. 


Implementation Goals: 


Goal 1: Academic Vocabulary Pre-test and Post-test 
Using the word map format, administer a pre-test and post-test over at least five new academic 
vocabulary words in a unit of study of your choosing. 


Goal 2: Implement Two “Something Old, Something New” Vocabulary Strategies 
BEFORE, DURING, and AFTER a reading assignment 

Develop a series of class lessons where you teach academic vocabulary using the “Something 
Old” strategy, Semantic Feature Analysis (SFA) AND the “Something New” strategies, 
Concept Maps (including the 6 Degrees of Separation Game) in a unit(s) of your choosing. 
You must have at least one vocabulary learning activity that occurs BEFORE students read your 
assignments, DURING the time students read your assignments, and AFTER students read your 


assignments. Note: Both of these strategies must be used somewhere in your lesson plans. Be 
sure to complete a GRID like the one below showing which vocabulary-learning activities you 
have selected for BEFORE, DURING, and AFTER students read the required text(s). 


STRATEGIES | UNIT SPI | DIFFERENTIATE | INTEGRATED GROUPING ASSESS- 
Which What is the INSTRUCTION PROJECT PLAN | What grouping MENT 
vocabulary unit or What are the multi- (IPP) strategies will What do you 
strategies will topic area level materials you will Explain how this you use with hope to see 


you use you will 
BEFORE, use? 
DURING, & 

AFTER the 

reading 

assignment? 


Pete | | | |__| | _ 
Reading the Text 
DURING the 
Reading of the 
Text 
Ee 
the Text 


use for Struggling unit might be your students? students doing 

Readers? integrated with as they complete 
another core subject each strategy? 
class. 
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Evidence Guide- This semester the literacy coaches will use 
the following tool, or a similar version, to guide their 
observations. Be sure your lesson plans address these areas. 


Categories Evidence 
Introduce the Strategy 


Teacher Modeling of the 
Strategy(s) 


Guided Practice 


Independent Use of the 
Strategy 


Differentiated Instruction 
(i.e., small group, cooperative 
groups, students working in 
pairs) 


Revisit the Strategy (after a 
period of time to check for 
retention, e.g., 4 weeks later) 


Student Assessment (What 
you hope to see students doing 
as they use the strategy) 


1 Before 2 During 3 After 
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CAP #1: Teacher — Literacy Coach Conferences Documentation 


Vocabulary Instruction: BEFORE, DURING, AFTER Reading 


Teacher: 


Subject Area: School: 


Coach’s 
Signature 


Activity 


Attended CAP Modeling/Discussion Session led by the 
Literacy Coach(es) about implementing word maps for 
pre- and post-test vocabulary assessment 


Lesson Plan Discussed with Literacy Coach Prior to 
Teaching (Must include SFA and 6 Degrees of Separation 
Game at least one time each in the unit) 


Literacy Coach Observes Teaching Rehearsal 


Debrief with Literacy Coach/Revise Lesson Plan as 


Needed 


Performance Teaching Observed by Literacy Coach 


Final Debrief with Literacy Coach 


-< 
tH 
n 
Z 
‘o) 


EXTRA CREDIT OPTION: Was “Classroom Blogging” 
attempted in at least one class? 
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Memphis Striving Readers Project: MCLA Component 1 


Memphis Content Literacy Academy 
Instructor’s Outline 
Session 1 


“Something Old, Something New” 
INCREASING VOCABULARY KNOWLEDGE 
Semester 3, Fall 2009 


From the syllabus... 


Session/ Tentative Topics and Classroom Action Plans Assignments & Other Critical 
Date (week) (CAPs)* Information 


Session 1 “Something Old, Something New” 1. Identify a Unit of Study for 


CAP #1 (should be one 


September 1, 3 INCREASING VOCABULARY KNOWLEDGE coming up in a few weeks). 


PART |: Course Introduction & Syllabus Review 2. From the unit you have chosen 
for CAP #1, select five (5) 
PART Il: TEACHING TACTIC/CONCEPT “non-negotiable” 
Strategies to help students learn BEFORE-, DURING- words/concepts you feel all 
and AFTER reading academic text assignments your student must learn and 
bring these to Session 2. 
PART Ill: BUILDING VOCABULARY 
“Something Old”: Semantic Feature Analysis (SFA); 
Word Maps (as an “assessment tactic”) 
“Something New”: Concept Maps; 
6 Degrees of Separation Game 


PART IV: Introducing Classroom Action Plan #1 


College of Education Norms 
EE eee 


w | take 100% responsibility. 

w | seek equity of voice. 

wv Lam willing to talk about sensitive issues. 
w | listen for understanding. 


¥ lappreciate the strengths and contributions of 
others. 


vw | bring positive energy and encouragement to the 
class 


ws lama professional and my actions reflect that role. 
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Memphis Striving Readers Project: MCLA Component 2 
INSTRUCTOR NOTES SESSION SEQUENCE & 
NEEDED MATERIALS 
WELCOME & INTRODUCTION Materials Needed: 


1. New student folders to be completed with contact information. 


POWERPOINT SLIDE #1 
(FOLDER INFO.) 


2. Name cards and markers (if you feel they will be needed. 

3. Copies of the Fall 2009 course syllabus. 

4. Option: See video performance “Africa” explained below. This 
can be very motivational to the group discussion. 

Procedure: 

1. Welcome everyone back 

2. Ask participants to fill in the contact information in their 
folders as indicated in the first slide of the PowerPoint 
presentation. 
Discuss the Fall 2009 course syllabus and upcoming topics. 
Explain that a theme for this first semester is “Something Old, 
Something New.” We will be revisiting some strategies from 
last year and using them in new ways (something old), and 
well as trying our some new teaching strategies and tactics 


POWERPOINT SLIDE #2 (something new). 
(You might draw attention to the phrase on 

Slide 2, “Research for America’s Schools” and 
MCS has an opportunity to “teach the nation” means to be a community of scholars working together, and 
about good instructional practices) 


5. Focus on the class norms and emphasize and focus on what it 


we value equity of voice. What might these norms look like in 
our class? Can we create that same kind of feeling with our 
students? 
INTERNET VIDEO OPTION: If you can project the Internet in 
your classroom (using your computer and the projector), show 
the “Africa” video found on this website as an example of a 
“community” working together with their simple God-given 
gifts to create some wonderful: 


http://www.youtube.com/watch?v=yjbpwiqp5Qw&feature=relate 
d 


Note: This video features a group of singers called Perpetuum 


“AFRICA VIDEO” OPTION Jazzile from that performs jazz and popular music. The choir uses a 
(Example of a ‘community’ effort in jazz 


singing, Perpetuum Jazzile, from an Eastern wide spectrum of jazz styles, performing complex and dense 


Block country) harmonies. 
KEY QUESTIONS FOR OUR PPT Slide #3: Introduces Professor William Tate, former president 
DISCUSSION: of the American Educational Research Association 
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Memphis Striving Readers Project: MCLA Component 3 


Why are we here? 
Is there really a need for MCLA? 


If so, what is the need? 


PowerPoint Slides: 3-8 
In these slides we share with Cohort 2 
excerpts from Dr. Bill Tate’s keynote 
address to the first cohort of MCLA. They 
vividly explain the status of Memphis 
students, reasons for helping our students 


stay in school, and their earning potential in 


life if they graduate from high school. 


(AERA), the top research organization in the 
education profession. 


Dr. Tate gave this keynote speech to the FIRST 
COHORT of MCLA. 


PPT Slide #4: Discuss briefly the GRADUATION rates in large city 
school districts. For example, Detroit only graduates 
about 22% of its students! The best of the district’s 
notes graduate well less than 50% of their students. 


PPT Slide #5: This slide shows the dropout rates in MEMPHIS. 
Notice that the dropout rate reported for Memphis is 
only for seniors. MOST DROPOUTS OCCUR WELL 
BEFORE THE SENIOR YEAR IN MIDDLE SCHOOL 
AND EARLY IN HIGH SCHOOL. The Memphis dropout 
rate, therefore, is much higher (about 45-50% 
overall). WE ARE LOSING TOO MANY YOUNG 
PEOPLE... 


PPT Slide #6-7: A POWERFUL MESSAGE—Students who graduate 
high school on average earn over $1.2 million dollars in 
their life time, and about double that if they get a college 
degree! 


PPT Slide #8: “Good teaching matters” was Dr. Tate’s conclusion. 
Here he shares examples from research in the Dallas 
schools. 


BRIEF DISCUSSION 
about 
Dr. Tate’s presentation 


Ask your participants to discuss with an “elbow partner” the BIG 
IDEAS from Dr. Tate’s presentation. After a few minutes, ask people 
to share their observations with the class. 


OUR PLAN OF ACTION 
TO HELP STUDENTS SUCCEED: 


Increase KNOWLEDGE, CAPACITY, 
FIDELITY 


PPT Slide # 9: This slide points out that we have three main goals 
this year in MCLA: 


1. KNOWLEDGE: To increase our knowledge about key strategies and 
tactic s for planning instruction that will increase student 
learning in the core subject areas. 


2. CAPACITY: Capacity means “expertise.” Our goal is to improve 
even more on selected strategies we learned last year, 
and develop NEW expertise using research-based 
teaching strategies. 


A big part of our challenge is to move from outdated and 
ineffective “traditional” teaching to more progressive and 
research-based practices that are proven to help urban 
students. 


Some of our colleagues, when it comes to changing how 
we teach, might say “If it ain’t broke, don’t fix it.” Our 
response is, “If it ain’t broke, we BREAK IT!” That is, 
we are BREAKING AWAY from traditional practice so 
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Memphis Striving Readers Project: MCLA Component 4 


TACTICS 


MCLA CONTENT LITERACY MODEL 


VYGOTSKY 


we can more to more effective practices. 


Can we do this? As President Obama says, “YES WE 
CAN!” 


3. FIDELITY: This goal actually goes along with Goal #2 in that we want 
to improve how well we use our strategies. In research 
we Call this “fidelity of implementation.” 


Let’s review for a few minutes our TACTICS for 
improving our teaching. 


PPT Slide # 10: This slide shows our “Content Literacy Model” we 
have developed for MCLA. Note that the three (3) 
key strands we will focus upon are: 


Fluency (reading at appropriate speeds for the kind of 
materials students are asked to read for assignments) 


Vocabulary and Concept Development (the special 
language of our core subject area that helps students 
become “math literate, science literate, and so forth) 


Comprehension of content reading (because if 
students do not understand what they are reading in our 
core subject areas, they cannot succeed) 


KEY IDEA: All planning for instruction should 
include these three components... 


PPT Slide # 11: This slide re-introduces the theories behind our 
teaching choices. It offers a famous quote in the 
research world—“There is nothing so practical as a 
good theory” (Lewin, 1952). 


PPT Slide # 12: This slide has quotes from Vygotsky who really gives 
us a justification for our teaching profession. An effective 
teacher: 


Understands the ability of his/her students (learned 
through assessments), 


Knows from assessment “what the student is ready 
for next,” 


Offers instruction that allows students to use what 
they have mastered as a “bridge” to new learning. 


PPT Slide # 13: This slide summarizes Vygotsky’s zone of proximal 
development or ZPD. It shows that the “sweet spot” for 
teaching and learning lies between what a student can 
already do competently on their own, and that which is 
beyond their grasp even if they had help. Out goal is to 
always teach in our students’ ZPD. 
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Memphis Striving Readers Project: MCLA Component 5 


ZONE OF PROXIMAL DEVELOPMENT 


GRADUAL RELEASE OF RESPONSIBILITY 


BEFORE, DURING & AFTER 
(BDA) 


Teaching with 
VERVE! 


PPT Slide # 14: TACTICS: GOING FROM THEORY TO PRACTICE. In 
this section we briefly summarize two KEY TACTICS we 
will use all year: gradual release of responsibility and 
strategies we want students to use BEFORE, DURING, 
& AFTER reading a core subject area assignment. 


PPT Slide # 15: Discuss briefly this slide showing what teaching using 
a gradual release of responsibility looks like. Note the 
changing responsibilities of the teacher and student. 


Explain that, like last year, our Classroom Action 
Plans of “CAPs” will be structured to help us provide 
students with learning experiences using gradual 
release of responsibility. 


PPT Slide # 16-19: Introducing the idea of helping students begin 
using certain strategies BEFORE, DURING, & AFTER 
they read a subject area assignment. The next few 
slides explain the benefits to students and their learning. 


PPT Slide # 20-24: Introducing a new goal of increasing VERVE in 
the classroom and taking advantage of students’ 
preferred learning “style.” 


These slides share research on African-American 
students and how teachers should consider learning 
activities that take advantage of VERVE to create 
VERVE-ACIOUS classrooms. 


PPT Slide # 25-26: Introducing the idea of using communalism in our 
learning activities to encourage discussion among 
students about what they are learning—a Vygotskian 
belief we operationalize through CREDE tactics to 
create a community of learners with our middle 
schoolers. 


PPT Slide # 27: Concluding PPT slide. 
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Memphis Striving Readers Project: MCLA Component 6 


Do the impossible? Yes we can! Yes we 
WILL! 


STRATEGY OVERVIEW 
BUILDING VOCABULARY 
“Something Old” 
Semantic Feature Analysis (SFA); 
Word Maps (as an “assessment tactic”) 
“Something New” 
Concept Maps; 


6 Degrees of Separation Game 


CAP #1 
“Something Old” 


¢ Word Maps used for ASSESSMENT 
¢ Semantic Feature Analysis (SFA) 


“Something New” 


Tactic: BEFORE, DURING, AFTER 
Concept Maps 

6 Degrees of Separation Game 
OPTIONAL: Classroom Blogging 


Distribute and Discuss 


Distribute CAP #1 and allow students time to read it over. 


Then, ask students to “retell” the main expectations of CAP #1 to 


an “elbow partner” and write down any questions for the instructor. 


Have a whole group question and answer session to clarify the 
expectations. Be sure to remind them that we will have three class 
sessions to prepare for CAP #1 before they start working with their 


Literacy Coaches on their plans. 


About the key strategies in CAP 
#1... 


PowerPoint Slides: 28-36 


In these slides you will review the “Something Old” strategies first. 


Be sure to note the NEW uses (ASESSMENT for “word maps”) and 
THINKING (SFA- BEFORE, DURING, or AFTER students read) with 
these strategies. 


In the “Something New” strategies we introduce concept maps, 
and leave them wondering until next week about the “6 degrees 
game” and the classroom blogging strategy. 


HOMEWORK ASSIGNMENT: 


PowerPoint Slide: 37 


1. Identify a Unit of Study for CAP #1 (should be one coming up 
in a few weeks). 


2. From the unit you have chosen for CAP #1, select five (5) 
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Memphis Striving Readers Project: MCLA Component 7 


“non-negotiable” words/concepts you feel all your student 
must learn and bring these to Session 2. 


TIP: Next week we will show them (briefly) how to do a “content 
analysis” to determine the non-negotiable FACTS, CONCEPTS, and 
GENERALIZATIONS in the unit they have selected. You might want to 
briefly discuss this idea for them to consider when choosing their five 
words/concepts. These should be IMPORTANT concepts any educated 
middle schooler should know (instead of extremely unusual ones we all 
seem to forget...). 


Class Evaluation & Reflections 


NOTE: IF YOU HAVE NEWCOMERS TO MCLA, YOU WILL NEED TO 
EXPLAIN HOW WE EVALUATE EACH CLASS MEETING. 


Class Evaluation- Distribute the class evaluation sheet to all students 
and ask them to complete it and place in the envelope provided. They 
should not put their names on the evaluation. 


Reflections- Ask students to write 3-4 sentences in which they reflect 
upon what they learned and did in today’s class session. They should 
place their reflection in their folder and drop it off on the way out. This is 
their “ticket out” each class session. 


GOOD LUCK WITH YOUR CLASS!!! 


MY REFLECTIONS ON THIS CLASS SESSION: 


Signature/Date 
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Memphis Striving Readers Project: MCLA Component 8 


Classroom Action Plan (CAP) #1 — Fall 2009 


Vocabulary Instruction: BEFORE, DURING, AFTER Reading 


Name School 


Subject Area/Grade Level Date Assigned: September 1/3 


Due: At your class meeting on September 29/October 1 


Directions: Develop lesson plans and execute your CLASSROOM ACTION PLAN (CAP) for 
your academic classes. Note: Please have your Literacy Coach(es) sign and date each stage of 
your CAP implementation. 


Implementation Goals: 


Goal 1: Academic Vocabulary Pre-test and Post-test 
Using the word map format, administer a pre-test and post-test over at least five new academic 
vocabulary words in a unit of study of your choosing. 


Goal 2: Implement Two “Something Old, Something New” Vocabulary Strategies 
BEFORE, DURING, and AFTER a reading assignment 

Develop a series of class lessons where you teach academic vocabulary using the “Something 
Old” strategy, Semantic Feature Analysis (SFA) AND the “Something New” strategies, 
Concept Maps (including the 6 Degrees of Separation Game) in a unit(s) of your choosing. 
You must have at least one vocabulary learning activity that occurs BEFORE students read your 
assignments, DURING the time students read your assignments, and AFTER students read your 


assignments. Note: Both of these strategies must be used somewhere in your lesson plans. Be 
sure to complete a GRID like the one below showing which vocabulary-learning activities you 
have selected for BEFORE, DURING, and AFTER students read the required text(s). 


STRATEGIES | UNIT SPI | DIFFERENTIATE | INTEGRATED GROUPING ASSESS- 
Which What is the INSTRUCTION PROJECT PLAN | What grouping MENT 
vocabulary unit or What are the multi- (IPP) strategies will What do you 
strategies will topic area level materials you will | Explain how this you use with hope to see 


you use you will 
BEFORE, use? 
DURING, & 

AFTER the 

reading 

assignment? 


fRewmenerae| | | PF 
Reading the Text 
DURING the 
Reading of the 
Text 
fer] | EP 
the Text 


use for Struggling unit might be your students? students doing 

Readers? integrated with as they complete 
another core subject each strategy? 
class. 


© 2009 Memphis Content Literacy Academy/Striving Readers Instructor’s Outline 
All rights reserved. Session 1, Fall 2009 


Appendix N. MSRP Evaluation Report, Year 4 March 1, 2011. Page 9 of 10 


Memphis Striving Readers Project: MCLA Component 9 


Evidence Guide- This semester the literacy coaches will use 
the following tool, or a similar version, to guide their 
observations. Be sure your lesson plans address these areas. 


Categories Evidence 
Introduce the Strategy 


Teacher Modeling of the Strategy(s) 


Guided Practice 


Independent Use of the Strategy 


Differentiated Instruction (i.e., small 
group, cooperative groups, students 
working in pairs) 


Revisit the Strategy (after a period 
of time to check for retention, e.g., 4 
weeks later) 


Student Assessment (What you hope 
to see students doing as they use the 


strategy) 
1 Before 2 During 3 After 
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Memphis Striving Readers Project: MCLA Component 10 


CAP #1: Teacher — Literacy Coach Conferences Documentation 


Vocabulary Instruction: BEFORE, DURING, AFTER Reading 


Teacher: 


Subject Area: School: 


Coach’s 
Signature 


Activity 


Attended CAP Modeling/Discussion Session led by the 
Literacy Coach(es) about implementing word maps for 
pre- and post-test vocabulary assessment 


Lesson Plan Discussed with Literacy Coach Prior to 
Teaching (Must include SFA and 6 Degrees of Separation 
Game at least one time each in the unit) 


Literacy Coach Observes Teaching Rehearsal 


Debrief with Literacy Coach/Revise Lesson Plan as 


Needed 


Performance Teaching Observed by Literacy Coach 


Final Debrief with Literacy Coach 


EXTRA CREDIT OPTION: Was “Classroom Blogging” 
attempted in at least one class? 


< 
ies) 
n 
Z 
e) 
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Coaching Daily Activity List 


Coach: 
Date: 
School Site: 


During Class Time 

* Observed teachers (provided onsite assistance through observation coaching) (#1) 
Demonstrated/Modeled CAP Lessons with MCLA participants and students (#2) 
Videotaped teachers (#3) 
Assisted teachers in other capacities (team taught, provided instructional or admin support) (#4) 
Other. 


Helped Teachers Prepare for Class (Instructionally) (#5) 
Make/wrote teacher-requested lessons, or created lesson plans 


¢ Gathered materials for teachers’ lessons 
¢ Make/wrote CAP lessons 
¢ Other 
Trained or Met with Teachers (#6) 
* Conferenced with teachers (e.g., reviewed CAPs, held planning mtgs, trained in use of CRC) 
¢ Gave feedback/support for teachers completing CAP 
¢ Provided individual professional development as needed to MCLA participants 
¢ Other 


Attended Coaching Professional Development (#7) 
¢ Participated in MCLA team planning/professional development events (off site), mentor mtgs, other 


¢ MCLA events, curriculum and instruction coach meetings 
¢ Read research and standards for Reading Specialists and Coaching 
¢ Other 


Performed coaching Administrative Tasks (related to MCLA) (#8) 
¢ Maintained/managed the Curriculum Resource Center (CRC) 
¢ Ordered supplies 
¢ Scheduled meetings, provided teacher with materials/supplies, emailed/corresponded, photocopied 


Performed Non-MCLA School-related Tasks (#9) 
¢ Assisted with TCAP activities or other (non-ITBS) testing, served as a substitute 
¢ Attended faculty meetings, attended rallies, homecomings, assemblies, math and science nights, worked 
in bookstore, etc. 


Performed MCLA-related School Tasks (#11- not #10) 
¢ Met with Instructional Facilitator/PDSCC 
¢ Visited with principal or other administrator to inform them of teacher needs 
¢ Helped with teacher MCLA recruitment 
¢ Other 


Striving Readers Evaluation Tasks (#10) 
¢ Prepared ITBS, assisted with Read 180 randomization, participated/collected surveys/interviews, assisted 
with the accuracy of data, met with RBS/Edvantia, etc. 


Conducted MCLA evening course tasks (#12) 
¢ Worked with lead MCLA instructors to deliver weekly course content 


99 = Other, fits no category 
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Analysis of MCLA Teachers’ Weekly Implementation of Literacy 
Strategies in Fall 2009 


As part of the evaluation of the Memphis Striving Readers Project (MSRP), RBS 
examined how often MCLA teachers used specific literacy strategies and whether 
content area teachers selected different strategies for implementation. In fall 2009, 
RBS collaborated with MCLA instructors to administer a Weekly Implementation of 
Literacy Activities (WILA) checklist to teacher participants. Survey items were culled 
from the Innovation Configuration Map developed by the MSRP grant team and 
reflected some of the activities that MCLA developers identified as crucial to successful 
classroom literacy integration. Each week, teachers were asked if they had engaged in 
the following eight activities in the past seven days: 


Put students into cooperative groups with assigned roles 

Informally assessed their students’ use of an MCLA strategy 

Pre-assessed their students’ content knowledge 

Met with literacy coach at school 

Formally assessed their students’ use of an MCLA strategy 

Met with grade-level, content-area colleagues during the school day to integrate 
literacy instruction into lessons 

* Received feedback from an administrator with regard to literacy integration 

e Used an MCLA strategy in their classroom in the past week 


The first WILA survey was administered during the third week of the MCLA class 
to allow time for exposure to, and adoption of, the new strategies. Participants 
completed the weekly survey six additional times in their content-specific MCLA class 
(i.e. social studies, science, mathematics, and English/language arts, or ELA). 
Instructors collected the surveys and returned them to RBS for data entry and analysis. 


All 93 (100%) MCLA participants in the fall completed at least one survey over 
the six-week period, however, most respondents completed five (32.3%), six (22.6%), or 
seven (11.8%) checklists over time. One-third (33.3%) of respondents completed four or 
fewer checklists. Of all the respondents who completed at least one survey, 28 (30.1%) 
participated in the ELA content class, 23 (24.7%) were in the mathematics class, 30 
(32.3%) were in social studies, and 12 (12.9%) were enrolled in the science MCLA 
course. 


The analysis of checklist responses addresses the following research questions: 


¢ To what extent do teachers report engaging in literacy-related activities in the 
past week? With what frequency do teachers report performing the activities? 


¢ To what extent is there an increase in the frequency of activities over time? 
¢ Are there differences in activity engagement by content area? 


RBS measured the change in respondents’ reported engagement in activities 
over the nine-week course, and restricted the analysis to the responses of 77 
participants who completed checklists frequently enough to allow for a comparison of 
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results at three different time points. More specifically, RBS analyzed the responses of 
teachers who completed checklists at either weeks three or four (baseline), five or six 
(midterm), and weeks eight or nine (follow-up). 


Overall, the analysis found the following: 


¢ Most respondents in fall 2009 reported a very similar number and type of 
literacy activities each time they completed the checklist 


¢ No meaningful differences emerged among teachers by content areas in the 
number or type of activities reported 


¢ Teachers were least likely to report having received feedback from an 
administrator in the past week: 14.3 percent of the 77 respondents reported 
receiving feedback at baseline, 23.4 percent received feedback at midterm, 
and 24.7 percent received feedback at follow-up 


Teachers’ Self-Reported Literacy Activity 


Table 1 summarizes the number and percentage of the 77 respondents who 
reported that they engaged in activities in the past seven days at baseline, midterm and 
follow-up. At baseline, respondents were most likely to report that they had put 
students into cooperative groups with assigned roles (83.7%), that they had informally 
assessed their students’ use of an MCLA strategy (81.4%), and pre-assessed their 
students’ content knowledge (76.7%). Receiving feedback from an administrator was 
the least frequently activity reported by teachers at each of the three time points. 


RBS conducted an analysis of variance (ANOVA) to determine whether the 
average number of activities reported by each teacher (between zero and eight) was 
statistically different at the three different time points. The test revealed no statistical 
difference in the average number of activities reported by the 77 teachers at the 
different time points. In other words, there was not a significant change over time with 
regard to the aggregate number of activities reported, in either direction (significance of 
the F-statistic was .712). 


Overall, there were no significant changes in the teachers’ reported literacy 
activities emerged over time. Results indicate that only two of the eight activities were 
reported at significantly higher rates over time: meeting with literacy coach (p<.05) and 
pre-assessing the student content knowledge (p<.05). However, a detailed analysis of 
the distribution of responses suggested interesting patterns across the individual 
teachers. For example, most teachers reported a similar number of literacy activities 
each time they completed the checklist. Some teachers consistently reported a low 
number (between 0 and 4) of activities; a second group reported a medium number (3 
to 5) of activities; a third group consistently checked off most of the activities (6 to 8). 


1RBS chose this approach because restricting the analysis to checklists completed at 
weeks four, six, and eight by the same individual limited the sample size to 46 teachers. 
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Only a small number of respondents submitted checklists that varied noticeably in the 
number of activities reported. 


Table 1 
MCLA Respondents’ Reported Engagement in Literacy Activities in the Past Seven Days 
Fall 2009 (N = 77) 


Baseline Midterm Follow-up 
# % # % # % 
Put students into cooperative groups with assigned roles 56 72.7 50 64.9 50 64.9 
Informally assessed their students’ use of an MCLA strategy 52 67.5 50 64.9 49 63.6 
Pre-assessed their students’ content knowledge 54 70.1 43 55.8 58 75.3 
Met with literacy coach at school 34 44.2 57 74.0 58 75.3 
Formally assessed their students’ use of an MCLA strategy 20 26.0 27 35.1 58 75.3 
Met with grade-level, content-area colleagues during the 
school day to integrate literacy instruction into lessons 39 50.6 42 54.5 42 54.5 
Received feedback from an administrator with regard to 
literacy integration 11 14.3 18 23.4 19 24.7 
Used an MCLA strategy in their classroom in the past week 44 57.1 46 59.7 42 54.5 


To investigate the extent to which teachers tended to report the same number 
(and type of) of activities across the semester, RBS counted the number of teachers 
who provided three or more identical ratings on their weekly surveys. Of the 62 
teachers who reported five or more total activities (the maximum was seven), 30 (48%) 
had identical responses at least three times over the course of the survey period. Since 
teachers were asked to complete the checklist weekly, it is not surprising that individual 
responses were stable over time. The reader is encouraged to consider data from other 
sources before drawing inferences based solely on these data. 


Content Area Differences 


RBS analyzed responses to the question asking respondents if they had used an 
MCLA strategy in the past week; Chi Square findings show no differences by MCLA 
professional development content area in self-reported literacy strategy use. More 
specifically, no significant differences emerged at baseline, midterm, or follow-up among 
the 77 teachers with matched reports by content area. 


A review of the type of activities reported at the three time points showed no 
differences by content area, except in two areas: at midterm, a smaller proportion 
(35.7%) of mathematics teachers than other teachers had met in the past week with 
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colleagues to integrate literacy into class (Chi Square = 1.00, df = 3, p=.018). 
Approximately 63 percent of ELA teachers, 42.9% of social studies teachers, and all but 
one of the 11 science teachers reported having met with grade level colleagues during 
the school day to integrate literacy instruction into the lesson. At follow-up, the 
proportion of science teachers who met with a literacy coach in the past week (36.4%) 
was smaller (Chi Square = 1.34, df = 3, p=.004) than for ELA teachers (87.5%), social 
studies (85.7%), and mathematics teachers (64.3%). No other content area differences 
emerged across the three time points with respect to the type of activity reported. 


Specific Strategies Reported 


RBS also asked respondents if they used an MCLA strategy in the past week and if 
so, to provide the strategy’s name or a description. Teachers often listed more than one 
strategy that they had used in the previous week. Table 2 shows the number of times a 
strategy was reported across all 93 MCLA participants’ checklists, in descending order. 
The three strategies that appeared most often among the teachers’ responses were the 
Frayer model, choral reading, and the concept map. 


Table 2 
Individual MCLA Strategies Listed on Weekly Checklists, Fall 2009 (N = 93) 


Strategy Count Strategy Count 
Frayer model 70 Academic reading 1 
Choral reading 60 Achieve 1 
Concept map 55 Assessments 1 
SFA 51 Bingo 1 
Word wall 32 CAD 1 1 
THIEVES 23 Character Analysis 1 
Word map 20 Classroom blogging 1 
Six degrees of separation 11 Cooperative learning 1 
Graphic organizers 8 Cornell notes 1 
Starburst 8 Flow chart 1 
Analogies ri Fluency 1 
LEARN 7 Gallery walk 1 
Word sort 7 Grid analysis 1 
Blog 6 Hierarchy 1 
Reader's theater 6 Joint productive activity 1 
Echo reading 5 KWL 1 
Neurological Impress Method 5 Mathematics 1 
Spider map 5 Peer tutoring 1 
Thinking maps 5 Poem for two voices 1 
QAR 4 Read aloud 1 
Vocabulary 4 Scooping 1 
Bubble map 3 Semantics 1 
Question stems | Six-steps 1 
Venn diagram 3 Summary chart 1 
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Antiphonal reading 2 Taboo 1 
Cloze reading 2 Theater 1 
Fix-up 2 T-diagram 1 
SEARCHES 2 Tree map 1 
Verve-acious classroom 2 Unison reading 1 
Vocabulary map 2 Wicker writing 1 
Wonder wall 1 
Word cards 1 
Word game 1 
WWII 1 


Data source: Fall 2009 WILA 


Table 3 shows the number of teachers at midterm reporting specific strategies by 
content area. The number of teachers in each MCLA content area is: 24 ELA, 14 
mathematics, 28 social studies, and 11 science. The number of teachers reporting a 
strategy does not total the number of teachers in a particular content area because 
teachers sometimes wrote that they used more than one strategy in the previous week. 


Table 3. Specific Strategies Listed by Teachers at Midterm by Content Area 
ELA Frayer model 

Word Map 

Six Degrees of Separation 
Choral Reading 

SFA 

Graphic Organizers 
Hierarchy 

Word Wall, Word Game 
Frayer model 

Word Map 

Choral Reading 

Word Wall 

Concept Maps 

Frayer model 

QAR 

THIEVES 

Choral Reading 
Antiphonal reading 
Graphic Organizer (e.g., sunburst) 
SFA 

Flow Chart 

Concept Map 

Word Wall 

Cornell Notes 

Thinking maps 

Taboo 

Frayer Model 

Word Map 

THIEVES 


Math 


Social Studies 


Science 
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Name 


PLEASE CHECK THE BOX IF THE STATEMENT IS TRUE FOR YOU. 


IN THE LAST SEVEN DAYS, I HAVE: 


L 


Ooo oO Uo ou 


LI 


Put students into cooperative groups with assigned roles. 


Informally assessed my students’ use of an MCLA strategy. 
Pre-assessed my students’ content knowledge. 
Met with a literacy coach at school. 


Formally assessed my students’ use of an MCLA strategy. 


Met with grade-level, content-area colleagues during the school day to integrate 
literacy instruction into lessons. 


Received feedback from an administrator with regard to literacy integration. 
Used an MCLA strategy in a content class. 


If YES, Which strategy/ies? 


(If applicable) To what extent did you help students with the MCLA literacy 
strategy? 


__ I helped them a great deal 
__ IT helped them a little 
___I didn’t help them at all because they used it independently 


__ other: 
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RAYS MCLA Participant Survey 


Research for Better Schools (RBS) continues to evaluate the MCLA program. Please take a few minutes to 
complete this survey. Contact Ms. Kelly Feighan at RBS (215-568-6150, ext. 285) with any questions. 


1. School: O American Way O Corry O Hickory Ridge O Lanier 


2. Teaching area: O ELA O Mathematics O Science O Social Studies = O Other: 


3. Participated in MCLA for: O All 4 semesters O 2 or 3 semesters O 1 semester 


To what extent do you agree or disagree with the following statements? (Please check a bubble.) 


Strongly Not Strongly 
Participating in MCLA... Disagree Disagree Sure Agree Agree 
4. helped me integrate literacy activities more often into O O O O O 
my lesson plans than before I participated in MCLA. 
5: resulted in more conversations with colleagues about O O O O O 
integrating literacy into my content area. 
6. did not make much of a difference in my teaching. O O O Oo O 
7. caused me to use literacy strategies more effectively in O O O O O 
my content classes. 
; : 
8. helped improve my students’ understanding of course O O O O O 
content. 
9. was useful; however, competing demands prohibit 
O O O O O 
frequent use of the strategies I learned in MCLA. 


10. How satisfied were you with the amount of modeling of strategies (from beginning to end) in a typical 
MCLA evening class this past semester? 


O Not at all satisfied O Somewhat satisfied O Satisfied O Highly satisfied 


11. About how often did you “work with” your MCLA literacy coach this year? By “work with” we mean, have a 
meeting, discussed or created a CAP together, or collaborated in some other way. 


O Never O Rarely O Sometimes O Often O Every week 


Spring 2010 ] 
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12. To what extent do you agree or disagree that your students do the following, AS A RESULT OF 
YOUR MCLA PARTICIPATION? 


Strongly Disagree Agree Strongly 


Disagree Agree 
a. use MCLA strategies with my assistance. O @, O O 
b. work with peers to use MCLA strategies when directed to do so. O O O O 
c. use MCLA strategies independently when directed to do so. O O O O 
d. work cooperatively in small groups. O O O O 
e. assume responsibility for assigned roles within their small O O O O 


groups. 


13. Please check the response below that best represents how you feel about working with the 
MCLA literacy coach: 


Strongly Not Strongly 
Disagree Disagree Sure Agree Agree 
a. Overall, I found the MCLA coach’s advice to be helpful. O O O O O 
b. The MCLA literacy coach was willing to help me when I O O O O O 
requested help. 
c. I don’t think my MCLA literacy coach really understood O O O O O 


what it was like to teach the content I teach. 


14a. How often have you used the MCLA Curriculum Resource Center? (This is the area in your school that 
houses materials provided by the program. The resources may have been stored in the library, a coach’s 
office, or in another room). 


O Never O Once O 2-4 times O 5 -7 times OQ 8 or more times 


b. If you’ve used materials from the “CRC,” to what extent did they help students meet content area objectives? 


O- Did not help at all O Helped a little O Helped O Helped a lot 


c. To what extent did you select the CRC materials to help motivate students to read? 
O Notat all O Alittle O A great deal 


Spring 2010 9 


Appendix Q. MSRP Evaluation Report, Year 4 


March 1, 2011. Page 3 of 4 


15. Thinking about MCLA activities 4 your school, how often did your PRINCIPAL do the following? 


Never Every At least At least At least 
other once twice once 
month monthly _ monthly weekly 
communicate a belief that literacy instruction is 
important for improving student achievement in oO O O O O 
the content areas 
express the expectation that teachers work with 
literacy coaches to support classroom O © © O O 
implementation of MCLA strategies 
conduct walkthroughs to see my MCLA O O O O O 
implementation 
provide feedback regarding my implementation O O O O O 
of literacy strategies 


16. How often (if at all) during this school year have you used the following? (If you select “1 to 2 
days per week” or more often, please also indicate in what percentage of classes you used the item). 


A. Used the adopted textbook to help students read and learn content 


OAlImost never 


OAt least once per month 


Ol to 2 days per wk 


OAt least 3 days per wk 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 


B. Used materials from the MCLA “CRC” library (e.g., National Geographic, Picture It!, etc) 


O Almost never 


OAt least once per month 


O1 to 2 days per wk 


OAt least 3 days per wk 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 


C. Used any MCLA supplementary materials appropriate for students’ reading levels 


OAIlImost never 


OAt least once per month 


O1 to 2 days per wk 


OAt least 3 days per wk 


Spring 2010 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 
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D. Used materials from other sources (e.g., online, colleagues, libraries) 
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OAIlImost never 


OAt least once per month 


O1 to 2 days per wk 


OAt least 3 days per wk 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 


E. Used cooperative learning activities as part of lessons 


O Almost never 


OAt least once per month 


O1 to 2 days per wk 


OAt least 3 days per wk 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 


In what percentage of classes? 
O 25% 
O 50% 
O 75% 
O 100% 


Feel free to write any parting comments below about your time in MCLA. 


Spring 2010 


THANK YOU VERY MUCH! 
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Memphis Striving Readers Project: MCLA Classroom Observation Protocol 


Observer: Obs. Date: / / Class length: _—_—s minutes 

School: Teacher name/gender: Female Male 
Was pre-obs. interview conducted prior to observation? Yes No [Admin only: ID = ] 
Class grade & content area: 6th | 7th ae ELA Math SS Science Other 

Adult present other than the classroom teacher? LJ Yes No [If Yes, note his/her role: ] 
No. students in class 15 min. into observation: _—s_—sés«éd([# girls: =o # boys: ——s|_~—s # of non-black students: 
Textbook: Was it used in class? Yes LINo 


Observation Codes: Please record the MCLA strategy code, instructional code, and level of engagement code observed 
in 10-minute intervals. If no literacy strategy is used, write “NLS.” Ifno instructional code is appropriate, write “NIC.” 


Interval 1 Interval 2 Interval 3 Interval 4 


Interval start & end times > : - ; : - : : - : : - 


MCLA strategy(ies) 


Instructional code(s) 


Cognitive demand code(s) 


Level of engagement 

LE = low (280% of students off-task) 
ME = mixed 

HE = high (>80% of students on-task) 


MCLA Priorities: Check the box that best represents what you observed. Be sure to include in your fieldnotes an 
explanation of your selection. 


1) Degree of students’ independent use of MCLA strategies: 
Students exhibit, when appropriate, independent and integrated use of multiple strategies. 
Students can self-select a strategy and use it independently. 


Students demonstrate independent use of the strategy (without teacher or peer assistance) when the teacher tells 
them to use a strategy. 


Students can use strategies with peers (cooperative or collaborative use) when teacher tells them to use a strategy. 
Students are aware of the strategy, can somewhat use it but not without some teacher assistance or scaffolding. 
Students engage in text-based work without the use of strategies. 


2) Student roles and behaviors during cooperative learning activities: 

Students have assigned roles, carry out those roles, and exhibit behaviors consistent with class norms for 
cooperative learning activities (e.g., observing equity of voice, listening for understanding, offering positive 
feedback, appreciating contributions of others, etc.). 

Students have assigned roles but do not carry out roles. Students do exhibit behaviors consistent with class norms 
for cooperative learning activities (e.g., observing equity of voice, listening for understanding, offering positive 
feedback, appreciating contributions of others, etc.). 

Students are grouped for tasks but do not have assigned roles. Students exhibit some behaviors consistent with class 
norms for cooperative learning. 

Students do not have assigned roles and do not exhibit behaviors consistent with class norms for cooperative 
learning activities. 

There is no evidence that students are grouped in cooperative learning activities. Students work alone. 


© Research for Better Schools, 2009. *Select items were adapted from the CETP Classroom Observation tool developed by Lawrenz, 
Huffman, and Appeldorn at the University of Minnesota’s College of Education and Human Development (2002). 
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Memphis Striving Readers Project: MCLA Classroom Observation Protocol 


PLEASE REMEMBER TO EMBED ALL MCLA STRATEGY CODES, INSTRUCTIONAL MODE 
CODES, AND COGNITIVE DEMAND CODES IN FIELDNOTES AS APPROPRIATE. 


MCLA Strategy Codes 
APR Activate prior knowledge JU Journal or blog use RTH Reader’s Theater 
B_ Bubble or double-bubble map LM Leveled content materials RR _ Repeated oral reading 
CC Context clue M Mnemonic strategies RT a eine sunmnanang al 
guidance 
CR Choral reading/whole group reading MU Monitoring understanding SFA cr ae analyse mens, 
CT Connecting text to students’ lives OR — Oral retelling SGQ Students generating questions 
DI pute sphoe msucuonrelaied toe PB Paired or buddy reading SW Shared writing 
literacy strategy 
E Etymology PT  Preteaching vocabulary TA Think aloud 
i Previewing text (e.g., THIEVES, ; : 
G Glossary or dictionary use PV LEARN, and SEARCH) TPS Think-pair-share 
GO Graphic organizer Q —_ Questioning for focus/purpose TRA Teacher models/reads aloud passage 
: ; Question-answer relationships/ : : 
GR_ Retelling with graphics QAR ReQUEST WR_ Written retelling 
IW Interactive word wall use REF  Reflection/meta-cognition WS_ Word sorts 
Instructional Mode Codes 
AD Administrative Tasks I Interruption RT _ Reciprocal teaching 
A Assessment J Jigsaw SGD Small-group discussion 
CD — Class discussion LC Learning center/station SP Student presentation 
DP Dalland pratuee (On, paper vocally, L Lecture TM _ Teacher modeling (problem modeling) 
or computer) 
Demonstration (e.g., science Lecture with discussion/ whole- Teacher/instructor interacting with 
D ‘ LWD ‘ . TIS 
experiment) class instruction student 
HOA Hands-on activity/materials OOC Out-of-class experience Vv Visualization (picturing in one’s mind) 
Reading seat work (if in groups, Writing work (if in groups, add 
oe add SGD) ne SGD) 


Cognitive Demand Codes 
1 Remember — Retrieve relevant knowledge from long-term memory (recognize, identify, recall) 
2 Understand — Construct meaning from instructional messages, including oral, written, and graphic communication 
(interpret, exemplify, classify, summarize, infer, compare, explain) 
3 Apply — Carry out or use a procedure in a given situation (execute, implement, use) 


Analyze — Break material into its constituent parts and determine how the parts relate to one another and to an 
overall structure or purpose (differentiate, organize, attribute, outline) 


5 Evaluate — Make judgments based on criteria and standards (check, coordinate, monitor, test, critique, judge) 


Create — Put elements together to form a coherent or functional whole; reorganize elements into a new pattern or 
structure (generate, hypothesize, plan, design, produce, construct) 


© Research for Better Schools, 2009. *Select items were adapted from the CETP Classroom Observation tool developed by Lawrenz, 
Huffman, and Appeldorn at the University of Minnesota’s College of Education and Human Development (2002). 
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DRAFT 


Annotated Guide to the Memphis Striving Readers Abbreviated 
Classroom Observation Protocol (MSR-COP) 


The purpose of this guide is to provide details about the abbreviated classroom observation 
protocol (COP) designed by Research for Better Schools (RBS) for use in the federally funded 
Memphis Striving Readers Project (MSRP). Additional questions or comments about using the 
protocol can be directed to Dr. Jill Feldman or Ms. Kelly Feighan at RBS (215-568-6150). 


It is presumed that if you are reading this manual, then you are a field staff member/observer for 
the MSRP. As such, you will be assigned to observe classrooms in one of eight participating 
middle schools. Prior to conducting observations, RBS will provide you with the name and 
contact information of teachers whose classes you will observe. Where possible, RBS will 
forward teacher and/or schedules. Since the teacher must sign a consent form prior to the 
observations, please check with RBS to see if you should e-mail the consent form to your 
assigned teachers in advance of the visit. Once you have received the schedule and have 
confirmed that teachers are aware of the upcoming visit, please do the following: 


¢ call and/or e-mail the teacher to express gratitude for his or her participation 
* confirm the date and time of your visit 
* suggest a time and date for conducting the brief, five-minute pre- observation interview 


The pre-observation interview must be conducted prior to the observation so that you have basic 
information about the lesson and students you will observe. 


Pre- and post- observation interviewing 

As an observer, you will be assigned to observe a class taught by a current or former participant 
of the Memphis Content Literacy Academy, a professional development program designed for 
teachers that promotes literacy integration. Whereas MCLA once targeted only content area 
teachers, such as mathematics, social studies, science, and English/language arts (ELA), it has 
since been made available to all full-time teachers in the participating middle schools. 
Occasionally, a READ 180 teacher also teaches ELA, and the roster you are given may be 
unclear as to which content class you are observing, so confirm that the class you have been 
assigned to observed is not a READ 180 class. If you are assigned to observe a READ 180 
class, you must use the protocol that has been designed specifically for observing that 
program. Please contact Ms. Debra Coffey at RBS (215-568-6150, ext. 276) for the READ 180 
observation protocol. 


If there are a few minutes prior to the start of the class, confirm with the teacher the answers 
from the pre-observation interview you ideally conducted a day or two before by phone or e-mail 
(see Appendix). Please remember that it is inappropriate to observe without some basic 
information about the lesson/group of students that will be observed, so at the very least, try to 
conduct the pre-observation interview right before class on the day as your visit if you must. The 
observation should span the entire classroom period. 


At the end of each lesson as the change of classes occurs, ask questions from Form B, the post- 
observation interview guide; however, if the teacher does not have time to speak to you at that 


Research for Better Schools 1 
2009-10 
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DRAFT 


moment, ask if you may return during the planning period or another time during the day. The 
questions will only take about 5 to 7 minutes to answer. E-mails are a last resort. 


General Information 

Please script what you hear particularly when it refers to the use of text or literacy when 
conducting the ten-minute interval observations so that you can go back later and code them. It is 
important to take very detailed notes about the material being taught, with examples of problems 
written on the board, questions asked and responses, and sketches of graphic organizers. Write 
down direct quotes from teachers and students wherever you can to capture them most 
accurately. Again, while you may refrain from taking detailed notes about disciplinary or 
administrative issues, or instruction that is not text- or literacy-based, it is especially important to 


note any conversation or instruction that turns to literacy matters or the use of text. 


Take the first few minutes of class time to complete some information required on the 
observation protocol, such as the teacher’s name and grade level. Do not burden yourself with 
coding the protocol’s matrix; instead, focus on taking descriptive notes so that you can code 
your notes after the observation. Mark the time often enough so that you can determine the 
ten-minute intervals, but do not spend so much time marking minutes that curtail your ability to 
take detailed notes. 


Make sure that the start and end of each interval is clearly marked in your field notes. You need 
not code literacy strategies while you are busy taking notes in class. Focus more on what you 
see and hear, and less on a code (that you will assign later). This will require that you write 
down sufficiently detailed information. Type up or polish your fieldnotes as close in time to 
the completion of your observations as possible. Use time between classes to fill in gaps in 
your notes or to type them up. Memory decay increases with each passing hour! 


When you type up your fieldnotes after the observations, include the following at the end of 
each time interval: 


¢ MCLA strategy codes found on the front of the COP (e.g., choral reading) 
¢ Instructional Codes found on the back side of the COP (e.g., lecture) 

¢ Level of Cognitive Demand found on the back side of the COP 

¢ Engagement level found on the front of the COP 


Do NOT put this information in text boxes in a margin. Do NOT write up your notes in a 
table or chart. They should look like this: 


1:35 — “Okay, we need to expedite matters,” the teacher says. “The word of the week that you 


see on the wall is metamorphosis (she spells it aloud and sounds it out slowly). I want you to tell 
me an animal that goes through it.” [L/LWD, PT, IW] 


MSR-COP: Background information 
In addition to including your name and the date on the first page of the protocol, record the 


teacher’s full name after confirming it when you enter or before you leave (one way is to listen 
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for students calling the teacher by name). Do not assume that you are observing the teacher on 
the roster (even if your roster includes teachers’ names). Determine that he or she is not a 
substitute teacher. Note any other adults in the room, and inquire about their role when you can. 
Approximately 15 minutes into your observation, record the number of girls and boys in the 
class. Since the majority will be African American, record as best as you can the number of 
students who do not appear to be of African-American descent. Be sure also to check the grade 
and content level of the class you are visiting. Record the total length of the observation in 
minutes. Do not count lunch breaks that occur in the middle of the class period in the total 


length. 


COP Matrix 


The next section to be completed on the COP is a matrix that asks observers to record time 
intervals, and codes for MCLA strategies, instruction, cognitive demand, and level of 
engagement. The following section provides details on each of these dimensions. 


MCLA Strategy Codes 
APR| Activate prior knowledge | LM_| Leveled content materials RT | Retelling/summarizing with gu 
CC_| Context clue M Mnemonic strategies RTH) Reader’s Theater 
Choral reading/whole grou 
CR_ | reading MU _| Monitoring understanding SC_| Scooping 
Semantic feature analysis, map 
CT_ | Connecting text to students’ ] OR | Oral retelling SFA| word grid 
Direct, explicit instruction 
DI_| related to a literacy strateg} PB Paired or buddy reading SGQ Students generating questions 
E Etymology PT Preteaching vocabulary SW | Shared writing 
Previewing text (e.g., 
FM | Frayer Model PV T.H.LE.V.E.S/S.E.A.R.C.H.) TA | Think aloud 
Glossary or dictionary 
G use Q Questioning for focus/purpose TPS | Think-pair-share 
Graphic 
GO _ | organizer/thinking map QAR | Question-answer relationships TRA| Teacher models/reads aloud pa 
GR_| Retelling with graphics REF | Reflection/meta-cognition WR | Written retelling 
IW_| Interactive word wall use | RAD | Radio Reading WS | Word sorts 
JU__| Journal or Blog use RR__| Repeated oral reading 


Glossary for “MCLA Strategies” 


Definitions for MCLA literacy codes are described below, in alphabetical order. 


Activating prior knowledge (APR): Any activity in which the teacher asks/discovers what 
students already know about a topic. This can include having the students complete an 
anticipation guide, brainstorm as a class, fill out a K-W-L, etc. Usually occurs before reading, 
but can also occur during reading in what are usually “teachable moments.” 


Choral reading/whole group reading (CR): This occurs when the entire class reads connected 
text in unison. Reading single words from a list does not count, as the text must be phrases, 
sentences, paragraphs, etc. Students are usually encouraged to use different vocal inflections. 


Researc 
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The passage may be read multiple times. Choral reading may take the following forms: (1) 
everyone reads in unison with or without the teacher leading; (2) antiphonal reading occurs 
where one group reads text together and another group follows by reading a different section; (3) 
one group reads aloud in unison, and another group (defined as two or more individuals) reads 
the same text selection again (“echo reading”). 


Connecting text to students’ lives (CT): The teacher or students relate the text material to 
current/commonly known events or experiences in students’ lives. Linkages that are made to 
previously acquired knowledge learned in class must also relate to life experiences in order to 
qualify. (Simply asking students, “Do you remember our discussion of that last time?” does not 
count as a connection made to their experiences). This strategy draws from students’ real-world 
experiences and/or knowledge and does NOT merely refer to something learned earlier in the 
year. Connections may be made about terms, text, concepts, or vocabulary (especially academic 
word wall examples). 


Context Clue (CC): The teacher directs the students to look in the text to infer the meaning of a 
word. This could include reading the rest of the sentence and figuring out what would make 
sense, pointing out a similar sentence or reference in the same passage, or directing students to 
look at an accompanying picture or chart. 


Direct, explicit instruction (DI): Deliberate, sequential instruction related to teaching literacy 
strategies. The steps include (1) introducing the strategy to students; (2) teacher modeling use 
of the strategy; (3) guided practice in which the students attempt using the strategy with a 
gradual diminishing of teacher support as students gain competence; and (4) independent use of 
the strategy by students for summative assessment purposes to ensure mastery has occurred. 
MCLA defines DI as “explain and model” (see appendix). If done in groups, add Small group 
discussion (SMG). DI cannot be coded without a corresponding literacy strategy during that 
interval. The key in capturing direct instruction is to identify the specific steps being used since it 
is rare to see all four steps occurring in the same lesson. 


Etymology (E): The teacher discusses the history or origin of a word. This often involves 
identifying bases, or common prefixes and suffixes and their meanings. 


Frayer Model (FM): This type of graphic organizer, an adaptation of concept maps, includes 
the concept word, definition, characteristics of the word, examples, and “non-examples” of the 
chosen concept word. See the appendix for an example of a Frayer model. 


Glossary or dictionary use (G): Students look up unfamiliar words as they are reading (or 
before or after reading) to find the definition. They can use a glossary provided by the teacher, a 
glossary in their textbook, a separate dictionary, or an online dictionary. Glossary/dictionary use 
can occur individually or in groups. 


Graphic organizer (GO): A general term for a form of “mapping” or pictorial representation of 
how concepts/facts in text are connected and organized. Examples of graphic organizers include 
story maps, semantic maps, bubble maps, K-W-L, character maps, cause-and-effect maps, 

problem-solution frames, Venn diagrams, and timelines. Graphic organizers might also be called 
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thinking maps (see appendix for examples). Can be done by students at desks or in groups, or 
by whole class using the blackboard/whiteboard, overhead projector, or other means of 
technology. 


Interactive word wall use (IW): Also called “academic word wall.” The word walls should be 
clearly marked, posted, visible, or presented to students so they have access to the words during 

related activities, which may include vocabulary games such as hangman or “password.” Words 
should be posted on the wall and easily visible. Use of the word wall would include adding new 
words or overtly referring to posted words. 


Journal or classroom blog use (JU): Students may write in journals or blogs during class. 
Although journaling is no longer taught in MCLA, you may see it in action. Journal writing is a 
type of routine where students reflect on content; it is an intellectual diary, rather than random 
paragraphs recorded on a piece of easily discarded paper. A journal is usually a discrete place or 
repository such as notebook. In classroom blogging, readers are able to post comments and have 
online chats facilitated by a host. MCLA participants will learn how classroom blogging can be 
used without technology by recreating the blog with large sheets of paper, sticky notes, and 
markers. 


Leveled content materials (LM): These are materials written on a variety of reading levels in 
order to help students with varying abilities gain access to information and increase reading 
skills. All students in the class will not be reading an identical passage if leveled materials are 
used. You may need to ask the teacher if this strategy has been used. The main source for 
leveled reading material in MCLA is the National Geographic series. Look for them in class. 


Mnemonic Strategies (M): Any strategy designed to aid memorization of a word. It can include 
making up a rhyme about a word, associating it with a familiar sound or rhyming word, or 
designing an acronym (such as HOMES for the Great Lakes). The strategy often has little to do 
with the meaning of the word itself. 


Monitoring understanding (MU): Here, the teacher gauges student understanding of text by 
asking specific questions regarding material and by encouraging students to monitor their own 
understanding of what is being taught. The teacher may ask open-ended questions such as “What 
do you make of that paragraph?” or “What did you get from that poem?” The cognitive level of 
his/her questions should be fairly high, rather than simple yes/no questions about facts. The 
teacher may also instruct students to use meta-cognitive strategies to gauge their own 
understanding of text (e.g., “Do not forget to use your fix-up strategies”). Overall, monitoring 
understanding of text can occur during and after reading. It does not include general strategies 
like asking if “there are any questions.” 


Oral retelling (OR): A post-reading strategy where students summarize or recap what was read. 
In some instances, students take turns summarize one of the sections (i.e., one student will retell 
the first section while the other student listens), then they will reverse roles for an oral retelling 
of the second part of the passage. After each retelling is done, the “listener” should ask the 
“reteller’ questions about anything gleaned directly from the passage. If a graphic organizer is 
used, code GO. 


Research for Better Schools 5) 
2009-10 


Appendix R-2. MSRP Evaluation Report, Year 4 March 1, 2011. Page 7 of 20 


DRAFT 


Paired or buddy reading (PB): Students are paired and both have copies of the text. One 
student will read aloud as the other follows along, and then they switch roles. Students may rate 
each other’s reading or time each other with a stopwatch. The passage may be read several 
times. Paired reading begins with one student assuming the role of reader and the other child 
assume the role of listener, prompter, and responder. The listener’s role is as important as the 
reader’s as he/she should provide supportive prompts and feedback when needed. Once one child 
in the pair finishes reading, the roles reverse. 


Preteaching vocabulary (PT): Pre-teaching of vocabulary involves the discussion of a word and 
its meaning before it is encountered in text. It can involve discussing examples of the word in 
various contexts and activating prior knowledge. Pre-teaching can occur along with other 
strategies. 


Previewing text (PV): This is generally a prereading strategy designed to increase 
comprehension. Students work through the text, identifying headings, titles, first sentences in 
paragraphs or sections, visuals (charts, graphs, pictures, etc.) and captions, summaries, etc. 
Students might create questions from this previewed material and answer them as they read the 
text. Includes T.H.I.E.V.E.S (during which students preview Titles, Headings, Introductions, 
Every first sentence, Visuals and vocabulary, End-of-chapter questions, and Summaries) and 
S.E.A.R.C.H.E.S., a similar technique adapted for use with word problems in mathematics. The 
T.H.IE.V.E.S. and S.E.A.R.C.H.E.S. strategies may occur either before or during reading. 


Questioning for focus/purpose (Q): Open-ended questions are used to enhance reading 
comprehension. It can include the teacher asking thoughtful questions about the passage in order 
to help the students understand the author’s point of view or the material being taught. This does 
not include basic knowledge questions. This is often, but not always, a pre-teaching strategy. 


Question-Answer Relationships (QAR): A strategy that can be used to increase comprehension 
during or after reading or to develop test-taking skills. There are two categories of QAR: Jn the 
Text and In my Head. These two categories are divided into four QAR types. Students are taught 
to identify the types of questions they are being asked, 1.e., whether the answer can easily be 
found in the text, in words that are both close to and similar to the subject of the question (“Right 
There”); whether the student will need to think about and find information included throughout 
the text to answer the question (“Think and Search’); whether the student would need to 
combine his or her prior knowledge with information from the text (“Author and You”); or 
whether the student will have to answer the question entirely from his or her prior knowledge 
(“On my Own’). The teacher and students may use a table to organize their thoughts. 


In the Text In my Head 
Right There Author and You 
Think and Search On my Own 


Radio Reading (RAD): Type of reader’s theater exercise in which a student reads aloud from a 
“script,” drawn from newspapers, magazines, or any print source that can be converted to a news 
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story. One students acts as a news anchor, and others act out and read the roles of other news 
journalists. 


Reader’s Theater (RTH): Fluency activity in which students read from a script, story, or poem 
and then share their oral reading with classmates and selected audiences. Students do not 
memorize the text or use elaborate props and instead present an interpretation of literature read in 
a dramatic style. Ifa story has been selected, students are assigned to read characters’ parts. If 
poems are selected for a reader’s theater, students may read alternating lines or groups of lines. 
Reader’s theater in the round, in which readers stand around the perimeter of the room and the 
audience is in the center surrounded by the readers, is a fun and interesting variation for both the 
performers and audience. 


Reflection/metacognition (REF): General term for any strategy during which students are 
actively stopping to examine whether they understand what they are learning/reading. Teacher 
may use “think-alouds” and students may discuss what they do when they don’t “get it.” 
Metacognitive strategies may use worksheets that direct students to use “fix up” techniques to 
improve their comprehension of text. This is a largely internal process, so take detailed notes 
about what is being said and done if you think you are observing use of this strategy. 


Repeated oral reading (RR): The same passage is read aloud multiple times (by teacher and/or 
students) while others follow along. May accompany other strategies. Repeated reading must 
involve extended text and not simply single words. 


Retelling/summarizing with guidance (RT): Students recap or paraphrase what was just read. 
This includes oral retelling, retelling using graphic organizers, and written summaries of silent or 
oral (verbal) reading. It can include an entire reading or just a paragraph. Basic summarizing is 
more general than retelling. 


Retelling with graphics (GR): A retelling strategy in which students use a graphic organizer to 
describe what they understood and remembered about a text. The organizer will usually include 
spaces for the main idea, key points, supporting arguments or details, but the shapes or structures 
of the organizers will vary. Retelling with graphics can include drawing a picture of what 
students understood from text. 


Semantic feature analysis, semantic maps, or word grids (SFA): A type of graphic organizer. 
A chart with topics listed in the left column and features in the top row. Each topic is marked 
(yes or no; plus or minus) to indicate whether or not the feature is present. See the appendix for a 
sample SFA. 


Students generating questions (SGQ): The students pose questions about the material read or 
learned, often using question stems provided by the teacher and/or whole class. This is an overt 
activity in which students are often in groups and told to formulate thoughtful questions, not a 
spontaneous activity inviting typical student questions. 


Research for Better Schools 7 
2009-10 


Appendix R-2. MSRP Evaluation Report, Year 4 March 1, 2011. Page 9 of 20 


DRAFT 


Shared writing (SW): teachers and/or students work together to create summaries about 
readings, or sometimes stories (most often in ELA classes), with the teacher or a student as the 
scribe. Can be done in pairs, small groups, or as a whole class. 


Think alouds (TA): The teacher describes their thinking process to model how a strategy is 
used. Literally walks students through (“Now I’m thinking that I don’t know the word, but I see 
this part looks familiar. ..”’) 


Think-Pair-Share (TPS): A cooperative learning strategy. The teacher poses an open-ended 
question to think about. Students first think on their own about their responses and share their 
ideas with a partner, before the pairs shares its ideas with the class. Only select if students are 
grouped in twos. 


Teacher models/reads aloud passage (TRA): The teacher reads or uses a model to read the 
passage fluently. Teacher should (but does not always) emphasize intonation and inflection as 
well as appropriate speed given the text demands. Students are usually asked to follow along and 
may re-read the passage afterwards. This refers to the reading of extended text (at least a partial 
sentence). 


Written Retelling (WR): This is a post-reading strategy that asks students to summarize what 
they have just read. There are two or three steps to this process that may be used (you may see 
any of these). The first step may be oral reading. Next, the student and/or class will use a 
graphic organizer (so code GO, too) to identify the facts, concepts, and generalizations from a 
passage. Then they will write a summary in paragraph form, expanding their generalization into 
an introduction, writing a paragraph for each concept using supporting facts, and restating the 
introduction in a conclusion. 


Word sorts (WS): Vocabulary activity in which students use new vocabulary (see 
interactive/academic word wall) to group, discuss, regroup, and discuss important vocabulary in 
a field of study. They sometimes may create a table (a graphic organizer) based on some 
common features of the target words. The sorting scheme is often related to the etymology or 
origin of the words or to their parts of speech. 


(Sample Word Sort) 
List of Words To Be Sorted (prefixes: astro-, bio-, chlor-, eco-, hydro-, hypo-, photo-) 


Word Sort 

astro- bio- chlor- eco- hydro- hypo- photo- 

astrology biodegradable | chloroform | ecosystem | hydrophobia | hypocrisy photography 

astronomer | biosphere chlorine economics | hydroplane_ | hypocondria | photosynthesis 

astronaut biography chlorophyll | ecology hydrology hypothesis photogenic 
biology hypoderm 
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Glossary for Instructional Codes 


Administrative Tasks (AD): teacher and students take care of nonacademic business, i.e., 
taking attendance, collecting homework, etc. 


Assessment (A): e.g., quiz, test. Assessment must be formal and at least 50% of the class must 
participate in the assessment. 


Class discussion (CD): Almost all student-to-student talk in a full-class setting. 


Drill and practice (on paper, vocally, on computer) (DP): Students participate in rote practice 
(such as math or vocabulary worksheets). If this involves writing, also code as WW. If this 
involves computers, also code as UT. 


Hands-on activity/materials (HOA): Students participate in an activity that involves 
manipulating materials. 


Interruption (I): e.g., visitor, unexpected announcements, student disruption. This does not 
include asides or reprimands to students. An interruption significant enough to warrant a code is 
one that has derailed instruction. Only code interruptions lasting more than several minutes. 


Jigsaw (J): A cooperative learning strategy. A topic is divided up into a few major areas or 
subsections. Students meet in expert groups to learn extensively about their own subsection. 
Then they join a group in which one expert from each subsection is represented and teach the 
others. Students must learn from each other to gain the whole picture. 


Learning center/station (LC): Students working at various stations related to particular topics. 
This may occur in elementary classrooms or in laboratory classes. 


Lecture (L): Teacher talks almost all the time. If students participate verbally, their interaction 
is minimal with questions and responses that are very short and/or obvious answers. 


Lecture with discussion/whole-class instruction (LWD): Teacher talks most of the time. This differs 
from lecture in that students participate by answering questions that generally require more than a one- 
word answer. This differs from class discussion in that there is almost no student-to-student 
communication. 


Out-of-class experience (OOC): e.g., field trips, interactions with other classrooms, concerts. 


Reading seatwork (RSW): Reading their textbooks or other written material including words or 
sentences on a blackboard or projected on a screen. If in groups, add SGD. 


Reciprocal teaching (RT): An instructional activity where students watch teacher model and 
then have the opportunity to develop skills during guided practice in four areas: prediction, 
questioning, clarification, and summarization. The teacher and students take turns assuming the 
role of teacher in leading the class discussion. As students acquire more practice with leading the 
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discussion, the teacher becomes a coach and allows the students to assume more responsibility. 
Here, the teacher steps out and acts as a coach. It is a very special case, does not happen often, 
and is built around literacy strategy learning. 


Small group discussion (SGD): Students (2 or more) engage in conversation with each other 
about subject matter in small groups. 


Student presentation (SP): e.g., student lecture, demonstration. This includes students going to 
the board to complete a problem or sharing their work (like a journal entry) from their seat. This 
does not include student responses to teacher questioning. 


Teacher/instructor interacting with student(s) (TIS): Teacher moving among individuals or 
groups of students and talking to them. 


Teacher modeling (TM): Teacher demonstrates or models with the whole class how to solve a 
new problem from start to finish. This also includes the teacher showing how something works 
or how to do something (formerly called “demonstration” where equipment was used). The 
teacher must actually do the activity while explaining it; not simply explain it. 


Visualization (V): Teaching students to create a mental image of information being read before, 
after, or while reading a text. 


Writing work (WW): Writing individually on worksheets, lab write-ups, journal entries, or 
other writing assignments. Can be combined with SGD. 


Level of Cognitive Demand 
What is “cognitive demand?” 


For our purposes in coding classroom observation data, cognitive demand is the level of effort 
exerted by students* in response to a learning activity or task. There are six major categories, 

which are listed on the MSP-COP (listed here), starting from the simplest behavior (level 1) to 
the most complex (level 6). The categories can be thought of as degrees of difficulties. That is, 
the first one must be mastered before the next one can occur. 


Cognitive Demand Codes 


1 = Remember Retrieve relevant knowledge from long-term memory (recognize, 
identify, recall) 


2 = Understand Construct meaning from instructional messages, including oral, written, 
and graphic communication (interpret, exemplify, classify, summarize, 
infer, compare, explain) 


3 = Apply Carry out or use a procedure in a given situation (execute, 
implement, use) 
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4= Analyze Break material into its constituent parts and determine how the parts relate 
to one another and to an overall structure or purpose (differentiate, 
organize, attribute, outline) 


5 = Evaluate Make judgments based on criteria and standards (check, coordinate, 
monitor, test, critique, judge) 


6 = Create Put elements together to form a coherent or functional whole; 
reorganize elements into a new pattern or structure (generate, 
hypothesize, plan, design, produce, construct) 


Citation: Anderson. L.W., & Krathwohl, D.R. (Eds.). (2001). A taxonomy for learning, 
teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: 
Addison Wesley Longman, Inc. 


Why are we bothering to code the level of cognitive demand? 

We’re concerned with cognitive demand because the goal of the program we’re evaluating is 
designed to result in teachers’ use of literacy strategies in ways that increase students’ 
achievement in reading AND in the core content areas. To effectively do this requires teachers 
to: 


1. learn about the literacy strategies (including how to implement each one) 


2. learn how and when to embed a particular strategy (or combination of strategies) into 
content-rich lessons in ways that 


3. increase students’ content-related knowledge 
It’s this third point that cognitive demand is designed to address. If a teacher successfully 
learned how to implement the strategies and used the strategies to design lessons that focused on 
key disciplinary concepts but did not require students to demonstrate more than a superficial 


level of understanding, it would be unlikely that increased achievement would result. 


Therefore, to code the cognitive demand of classroom observation data that you collect you will 
need to: 


1. have a solid understanding of each of the six categories 


2. be able to classify the cognitive demand exerted by students* in response to a learning 
activity or task 


*Note: It is possible for a teacher to design an activity or task with a high potential level of 
cognitive demand. However, we are coding the level of cognitive demand of students’ observed 
responses to the activity or task. For this reason, it is important when you are taking notes to 
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capture as much of the teacher’s verbatim instructions in setting up the activity as well as 
students’ verbatim responses. 


Codes for item “Student Engagement”: Only one engagement code should be recorded for 
each interval. If engagement varies, choose ME (mixed engagement). 


LE = low engagement, 80% or more of the students off-task 
ME = mixed engagement 
HE = high engagement, 80% or more engaged 


Rating Students’ Use of MCLA Strategies 


The last two items on the COP require that the observer use his or her best judgment in 
characterizing the level of students’ use of MCLA strategies. The instructions read, “Please put 
a check mark beneath the box which best represents what you observed in class.” The first item 
pertains to students’ independent use of strategies, while the second item refers to student roles 
and behaviors during any cooperative learning activities you may have observed. Please do your 
best in checking a box for both items based upon what you saw during class time. If there is no 
evidence of literacy strategies or cooperative learning activities, check the final box to the right. 


***T¢ is important to be familiar with the rubric and include in your notes a summary of 
why you came to your rating. 


Tips for Assigning Ratings to Students’ Use of Literacy Strategies 


1. Independent use of MCLA strategies 
Some strategies are designed for use with groups of students (e.g., choral reading, 
paired reading). The important things to focus on are: 
a. Who decides when/whether to use a strategy? 
i. If student(s) decide, select a or b, as appropriate 
ii. If teacher decides, select c, d, e, (or f), as appropriate 
b. Do students require help in order to use the strategy? 
i. Do they receive support from peers? (select d) 
ii. Do they require help from the teacher? (select e) 


*Note: Review intended model for gradual release of responsibility to students 
iii. Introduce 
iv. Model 
v. Guided practice 
vi. Independent use 


2. Student roles and behaviors 


Not all small group activities are instances of cooperative learning. In cooperative 
learning activities, students are assigned specific roles (e.g., “recorder,” “reporter,” 
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“getter,” etc.). In deciding how to rate the class you are observing consider the 
following: 
a. Assignment of Roles 
i. Do students have assigned roles? 
ii. Do students actually perform the role they were assigned? 
iii. Are students simply grouped but not assigned specific roles? 
b. Behavior 
i. Are students engaged in discussion about content? 
ii. Are students responsive to each others’ comments (or are they non- 
responsive or only respond to comments made by the teacher)? 
iii. Are there respectful interactions among students? 


When in doubt: 


o Describe in as much detail as you can: 
o What (if any) MCLA strategies were used? 
o Who decided whether/when to use the strategy? 
o Did students require help in order to use the strategy? 
° 


Where were students situated within the classroom (e.g., in rows facing front, in 


tables of 4, etc.)? 
Specific roles assigned to students. 
o Student behavior/interactions. 


Oo 
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Research for Better Schools, 2009 
Teacher Pre-Observation Interview Guide* 
Hello, this is from in ' . We’ve 


contracted with Memphis City Schools to conduct an independent evaluation of the Memphis 
Striving Readers Program. As part of this four-year evaluation, we are randomly selecting 
classrooms to observe. You are being asked to participate in this interview because one of your 
classes was chosen for observation. 


This interview should take no more than five minutes. Please answer the questions as best as 
you can. As you read in the informed consent form that you have signed, your responses will be 
kept confidential and you will never be identified by name when we report the results of these 
interviews. Also, this interview and the observation will be used for research only, never for 
professional or personnel evaluation. You are free to stop participating or withdraw at any time. 
Let me know if you would like to skip a question because you don’t know how to respond to it. 


May I start the interview now? Please think about the class you teach during period. 
[This will be mathematics, science, language arts, or social studies class]. 


1. What has this class been doing recently? /Probe: on what unit are you working? What 
instructional materials are you using? What, if anything, will you be asking the students 
to do during that class?] 


2. Where does lesson fit within the unit? 


3. What do you anticipate doing in class today (or on the day I will be observing)? /Probe: 
What do you hope students will learn as a result of the work you have planned? ] 


4. What is the content-related objective(s) of the lesson? (If teacher provides only a short SPI 
number, ask for specific detail about the goal(s) of the lesson). /Probe: What, if anything 
do you hope students will learn about MCLA strategies as a result of the work you have 
planned?] NOTE: If the pre-observation interview is NOT conducted, this question 
should be added to the post-observation interview. 


5. Is there anything in particular I should know about the students I will be observing (e.g., is 
it an honors class?) 


Thank you so much. See you on [observation date]. Please remember that I am coming to 
evaluate the MCLA program (the purpose of our study), not your performance. 
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DRAFT 


Form B 


RS 


Research for Better Schools, 2009 
Teacher Post-Observation Interview Guide* 
Thank you for allowing me to observe your class. This interview should take approximately 
five minutes. Please answer the questions as best as you can. Your responses will be kept 
confidential and you will never be identified by name when we report the results of these 


interviews. You are free to stop participating or withdraw at any time. If for any reason you 
would like to skip a question, just let me know. May I begin? 


*NOTE: If no pre-observation interview was conducted, begin with the following question: 


What was the content-related objective(s) of the lesson I observed? (If teacher provides only a 
short SPI number, ask for specific detail about the goal(s) of the lesson). /Probe: What, if 
anything do you hope students will learn about MCLA strategies as a result of the work you have 
planned? ] 
1. Were there any ways in which the lesson was different from what you had planned? 
2. What did the lesson tell you about what your students are learning, and still need to 
learn? [Probe: What did you observe during the lesson that makes you think this? How 
do you plan to further assess the students’ learning?] 


3. What, if anything, do you plan to change in response to what we observed today in class? 


4. What, if any, challenges have you faced using cooperative groups to actively engage 
students in this class? /Probe: How have you approached these challenges ?] 


5. For what purpose do you use cooperative groups? 
Important: If the teacher and/or students used a literacy strategy, ask the following: 
a. Have these students used the strategy in your class before today? 
b. What were you hoping to accomplish by using the strategy? 
c. How do you think it went? [Probe: What makes you think so?] 
d. How challenging was it for you to integrate that activity into this lesson? [Probe: 
What in particular was challenging? How did you address this? What would 


have made it less difficult? ] 


6. What is the next step for this class? 


Research for Better Schools 1) 
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Strategy Examples 
Excerpt from MCLA Session #8 in which DI is explained: 


---- Design a lesson plan(s) in which you EXPLAIN and MODEL TWICE how to complete the 
GRAPHIC ORGANIZER (G.O.) you have selected. Model the G.O. using text the 
students have already studied as an example, then move on to a second modeling where 
you use another text the students have already studied. Modeling twice helps to 
“marinate” them in the process. Be sure to include a handout you design that shows 
students a step-by-step process for completing a graphic organizer (you may want to 
consider revising the handout from this class on how to complete a Content Analysis). 
This is part of DIRECT INSTRUCTION that is important for student understanding. 


---- Design a GUIDED PRACTICE lesson plan(s) in which your students complete the 
GRAPHIC ORGANIZER you introduced in your MODELING sessions. This should be 
completed in pairs or in groups no larger than four students. 


Frayer Model 


(http://www.worksheetworks.com/miscellanea/graphic-organizers/frayer.html) 


Characteristics 


Definition 
glassy 


A solid made of clear colored 
atoms arranged in an brightly colored 
ordered pattern. evenly shaped 


patterened 
glimmer, sparkle 


Examples 
metals Non Examples 
rocks coal 


snowflakes pepper 
salt lava 


sugar obsidian 


Frayer, D., Frederick, W. C., and Klausmeier, H. J. (1969). A Schema for Testing the Level of 
Cognitive Mastery. Madison, WI: Wisconsin Center for Education Research. 
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DRAFT 


Example of a Reciprocal Teaching lesson plan: 
1. Put students in groups of four. 


2. Distribute one note card to each member of each group identifying each person's unique role. 

* summarizer 

* questioner 

* clarifier 

* predictor 
3. Have students read a few paragraphs of the assigned text selection. Encourage them to use 
note-taking strategies such as selective underlining or sticky notes to help them better prepare for 
their role in the discussion. 


4. At the given stopping point, the Summarizer will highlight the key ideas up to this point in the 
reading. 


5. The Questioner will then pose questions about the selection: 
* unclear parts 
* puzzling information 
* connections to other concepts already learned 
* motivations of the agents or actors or characters 
° ete. 


6. The Clarifier will address confusing parts and attempt to answer the questions that were just 
posed. 


7. The Predictor can offer guesses about what the author will tell the group next, or, if it's a 
literary selection, the predictor might suggest what the next events in the story will be. 


8. The roles in the group then switch one person to the right, and the next selection is read. 
Students repeat the process using their new roles. This continues until the entire selection is read. 


Excerpt from http://www.readingquest.org/strat/rt.html 


Bubble Map (B): Double-Bubble Map (B): 


delicious | ssa 


tabby, 


\ , \ / \ skittish be 
sweet fattening | | smooth | friendly 
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DRAFT 


Sample: Semantic Feature Analysis (S): 


U.S.A. Russia Australia Taiwan Philippines Indonesia Singapore 


Democratic 
gov't 


Population 
more than 100M 


Centrally 
Planned 
Economy 


+ 


+ + + + - - 
+ - - = 3 : 
+ . : : + + 


Sample Written Retelling completed outline (WR): 


Introduction 


Topic #1 (from 
graphic organizer) 


Supporting details 

Concluding sentence 
Topic #2 (from 
graphic organizer) 
Supporting details 
Concluding sentence 
Topic #3 (from 
graphic organizer) 
Supporting details 
Concluding sentence 


Topic #3 (from 
graphic organizer) 


Supporting details 


Concluding sentence 


Research for Better Schools 


2009-10 


Spiders are very interesting creatures, and are even scary to some people. The 
scientific name for spiders is “arachnids,” and they are insects. There are 37,000 
kinds of spiders. There are even songs about spiders we learn in school. In this 
report we will learn facts and misconceptions about this special living thing we 
learned in our book and on the internet. 


One misconception is that all spiders are poisonous. Some spiders are 
poisonous, or “venomous,” but not all spiders have venom. Spiders use venom 
to stun or kill creatures they want to eat. Of over 37,000 kinds of spiders, only 
about 25 have venom that can hurt humans. Two spiders in the U.S. with venom 
that can hurt humans are the black widow and the brown recluse, but no one has 
been proven killed in over two decades (20 years). 


Another misconception is that some spiders can be larger than a cat. Spiders 
come in many sizes. The largest is the Goliath bird eater tarantula. It is found in 
the rain forests of northeastern South America, and can be as big as a dinner 
plate It can grab birds from their nests! The smallest spider is from Borneo and 
is the size of a pinhead. So, there are no spiders larger than a cat, but they can be 
very large and also very small. 


One thing we learned is that different arachnids eat different things. Many 
spiders eat insects, but not all do. There are spiders who dine on birds, frogs, 
fish, lizards, and snakes. So it is not true that all spiders eat bugs! 


There are other things about arachnids, or spiders, that we still do not know. 
What is the largest spider in North America? Is it as big as the Goliath bird eater 
tarantula? We hope not. Also, is the silk spiders make all the same kind? How 
strong is their silk? Could you make clothes out of spider silk? We wonder 
where the name “arachnid” came from? And what about water spiders? Do any 
of them actually live under water? We still have a lot to learn about arachnids. 
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DRAFT 


Spiders, or arachnids, are very interesting insects. They come in many sizes, live 
on different things, and some are poisonous. We want to know more about this 


special creature. 


Conclusion 


From: Reutzel, D.R., & Cooter, R.B. (in press for fall 2007). Teaching children to read, 5“ ed. Upper Saddle River, NJ: Pearson 
Merrill Prentice-Hall. 
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Ry Teacher provides 
.@" content instruction 
oe only. 


Teacher provides 
% content instruction 
wv only. 


Teacher provides 
instruction without 
guided practice. 


Teacher uses 
continual teacher- 


directed whole-class 


instruction to guide 
students’ strategy 
SS application. 


xe Teacher relies 
primarily on whole- 
group instruction. 


(leacher introduces 
each strategy once 
3s but does not revisit 


presented. 


when new material is 


Teacher mentions the 
strategy but does not 
provide students with a full 
description of the purpose of 
the strategy and when it is 
to be used. 


Teacher makes passing 
reference to the strategy 
with no modeling provided. 


In providing instruction, 
teacher involves students in 
follow-up activities without 
feedback. 


Teacher provides 
opportunities for students’ 
independent practice but 
does not monitor students’ 
progress. 


Teacher differentiates 
instruction but does not use 


Teacher makes passing 
reference to previously 
taught strategies without 
providing opportunities for 
students to apply those 
strategies to new material. 


Teacher (1) names the strategy and 
(2) describes the purpose of the 
strategy and when it is to be used. 
Teacher activates students’ 
background knowledge and 
experiences to help them 
understand the strategy. 


In providing explicit and direct 
instruction, teacher occasionally 
models initial use of the strategies. 


In providing explicit and direct 
instruction, teacher occasionally 
involves students in guided practice 
activities and provides general 
feedback. 


Teacher provides opportunities for 
students’ independent practice and 


monitors students’ progress applying 


strategies to assess additional 
learner needs. 


Teacher differentiates instruction 
based on analysis of progress 
monitoring (e.g., small groups, use 
of technology, reteaching, use of 
curriculum resource center 
materials). 


Teacher occasionally revisits 
previously introduced literacy 
strategies as opportunities to apply 
them to new material. 


In providing explicit and 
direct instruction, teacher 
consistently models initial 
use of the strategies (e.g., 
think-alouds, questioning). 
In providing explicit and 
direct instruction, teacher 


consistently provides 
multiple guided practice 


activities using a variety of 


texts. Students receive 
relevant feedback with 


respect to their use of 
specific strategies. 


Teacher consistently 
revisits previously 
introduced literacy 
strategies as opportunities 
to apply them to new 
material. 
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(e.g., Evidence 
sheet, Observation 
notes or written 
notes, copy of class 


*Differentiated 
instruction or 
"learning style." 
Technology is not an 
essential tool. 
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Appendix T: Difference-In-Differences Estimations of the Impacts 
of the Whole-School Intervention 


This appendix presents the results from estimating the impact of MCLA exposure on students 
in the Cohort 2 schools using a “difference in differences” approach. It is included here to 
supplement the analyses presented in Section VI of the report. As with the OLS regression 
analysis and the analysis of simulated outcomes, the evaluation year (Year 2 or Year 4) was the 
“treatment” variable used to represent the difference between student achievement before and 
after the whole-school intervention. The dependent variables, however, are modeled differently; 
they were calculated as the difference between the ITBS Total Reading, Comprehension, and 
Vocabulary NCE scores obtained at the end of Years 2 and 4 and the appropriate baseline score 
for each of these. Thus, the dependent variables are represented as the difference between Y, 
(post-test) and Yo (pre-test), or Y; - Yo. 


Following the difference-in-differences approach used by Rosenshine (2003) to estimate 
effects of high-state testing and presented by Buckley and Shang (2003), RBS first modeled the 
equation for estimating the effects of MCLA on student outcomes first as a simple OLS 
regression model with no additional covariates and then modeled the equation in an expanded 
model that included two types of covariates: several student demographic characteristics (gender, 
free or reduced-price lunch status, English language learner, and African American), and three 
dummy variables representing the differences among the four schools. These equations are listed 
below: 


(1) Y1- Yo= 50 + 6;(Treatment Variable) 


(2) Yi- Yo= 50 + 6; (Treatment Variable) + 62(Female) + 63(Free or Reduced Lunch) + 
64(ELL) + 65(AfrAmerican) + d6(School A) + 67(School B) + 5g(School C) 


Equations (1) and (2) have been called in the methodological literature “a change score 
method.” It should be noted that if the pre-score was added as a covariate in Equation (2), that 
would be computationally equivalent to the OLS regression with posttest as a dependent variable 
presented in Section VI, because the estimated coefficient and standard errors for the treatment 
variable would be the same (Allison, 1990, p. 94). The change score method has been the 
subject of a broad debate in the methodological literature, which Allison (1990) summarizes in 
his article, and has been challenged with respect to its unreliability and sensitivity to regression 
toward the mean. However, evaluators decided to use this model and conduct the analyses, 
following Allison’s demonstration that those criticisms are not universally sustained and that 
there is merit to pursuing the “change score method” and comparing it to the OLS regression 
model with the posttest as a dependent variable. 


The results from equations (1) and (2) are presented in Table | and Table 2, respectively. 
The findings corroborate the conclusions evaluators reached after using the two approaches 
reported in Section VI. With the expanded model that accounts for the same covariates as the 
OLS regression and the simulated outcomes approach, the significance level and magnitude of 
effects on eighth-grade students’ ITBS Vocabulary and Total Reading and the seventh- and 
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eighth-grade students’ ITBS Vocabulary scores are similar to the results from Section VI. 
(Without controlling for demographic or school covariates, researchers using the change score 
method also found that the estimated effects of MCLA treatment on the eighth-graders’ ITBS 
Comprehension, and the seventh- and eighth-graders’ ITBS Reading and Comprehension change 
scores were significant.) This increased the evaluators’ confidence in the findings presented in 
Section VI, and creates a compelling case for using several different methodologies in order to 
confirm the rigor of estimates and the resulting conclusions. 


Table 1: Simple model — the only independent variable is MCLA treatment 


Change in Test Score MCLA treatment 
Grade from Pre to Post N effect t-test Significance 
6 ITBS Total Reading 1099 1.078 1.651 0.099 
ITBS Comprehension 1106 0.537 0.727 0.468 
ITBS Vocabulary 1119 1.288 1.675 0.094 
7 ITBS Total Reading 848 -0.560 -0.703 0.482 
ITBS Comprehension 892 0.071 0.082 0.934 
ITBS Vocabulary 854 -0.886 -0.878 0.380 
8 ITBS Total Reading 846 3.078 3.339 0.001 
ITBS Comprehension 852 1.404 1.423 0.001 
ITBS Vocabulary 864 4.215 4.075 0.000 
7 and 8 | ITBS Total Reading 1694 1.225 2.032 0.042 
ITBS Comprehension 1744 0.709 1.089 0.276 
ITBS Vocabulary 1718 1.632 2.248 0.025 
Table 2: Model with Covariates, Excluding Baseline Score 
Change in Test Score MCLA treatment 
Grade from Pre to Post N effect t-test Significance 
6 ITBS Total Reading 1099 1.092 1.666 0.096 
ITBS Comprehension 1106 0.538 0.727 0.468 
ITBS Vocabulary 1119 1.341 1.738 0.082 
7 ITBS Total Reading 848 0.067 0.081 0.936 
ITBS Comprehension 892 0.107 0.123 0.902 
ITBS Vocabulary 854 -0.263 -0.250 0.802 
8 ITBS Total Reading 846 2.838 3.199 0.001 
ITBS Comprehension 852 1.136 1.165 0.244 
ITBS Vocabulary 864 4.056 3.899 0.000 
7 and 8 | ITBS Total Reading 1694 1.109 1.819 0.069 
ITBS Comprehension 1744 0.615 0.935 0.350 
ITBS Vocabulary 1718 1.479 1.999 0.046 
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